Rocio: Thank you. So I am Rocío Carratalá from Spain from, and I’m currently an assistant professor.

Bilel: So, hello. Good afternoon. I’m, uh, Bilel Hadri computational scientist at the KAUST Supercompeting CoLab, and , I am the Reproducibility Initiative Chair for SC 22.

Le Mai: Hi, my name is Le Mai Weekly. I am a senior technical lead with the research applications team with Indiana University, and I am participating with the Reproducibility Challenge in the Student Cluster Competition and the special journal issue under the reproducibility initiative.

Nicole: Wonderful. Thanks for coming. So today we’re gonna talk about the reproducibility committee, which you are all a part of in one way or another. The reproducibility committee has been around for a while, so if you all would like to start with explaining your roles and how that contributes to computational reproducibility in general.

Bilel: To briefly summarize the, effort of the reproducibility at SC. So this effort started since, 2015. And it was led by Michela Taufer. And at that time there was like, if I recall correctly, like one paper that has shared like the, about the artifacts kind of brief description.

And, after seven years, quite a big committee with a lot of task and so thanks to many effort of all our predecessor. So now the reproducibility initiative has like major, like in three subtracts, the artifacts, description, and evaluation to award with the badges.

We have also the journal issue submission that are reviewing the student challenge that have selected one of these apps. And also last but not least, one subcommittee for Reproducibility Challenge that they select a previous paper from the last year and they selected for the student challenge.

So basically the effort spent for over three years. And now what is new? Like since 2021, there is an award for the best paper.

Le Mai: This year I’m doing the special issue for the reproducibility initiative. So, as Bilel was mentioning, this is like the last leg of that life cycle, three year life cycle of, a paper that goes into SC. So first of course, when you submit a paper to SC because of the reproducibility initiative you also have to submit an artifact descriptor, which will, be evaluated and then possibly get the badges and then maybe in the end get an award depending on what happens. Then, once a paper gets submitted, the reproducibility challenge selects a paper from the previous year and creates tasks for the student cluster competition.

To try to reproduce, results that they’re seeing in their paper. So the full circle comes in with a special issue where we, invite the best team, the top scoring reports from the previous student cluster competition, invite them to, shepherd them into bringing their reports that they wrote on the floor into journal quality.

And then inviting the author of that original paper to revisit their paper, with the perspective of these results that the students saw and extend their paper commenting on what the results of the paper saw so that paper from two years ago finally sees its end with students actually working on it and trying to reproduce their stuff.

Rocio: Yeah, so I’m co-chairing SC 22 Artifact Documentation and Artifact Evaluation Committee this year. And I think what this initiative has been focusing on evaluating all the technical paper artifacts provided so we can check if they are awarded with one, two, or three badges respectively correspond to first badge.

The artifact is available, so it has a DOI and it’s accessible. Second one, it’s functional, so you can download it, compile it, and use it. On third one, it’s, reproducible. It means that the results observed from the execution are those claimed in the paper they belong to.

Nicole: Great. Thank you guys for that. And thank you for the history there that I feel like that’s a really important start.

Bilel: We just have to give the credit to the people who have created this initiative.

Nicole: Yeah, so I guess I will introduce myself as well just to give you a motivation for why I put this together. One year I did the Student Cluster Competition. I, focused on the reproducibility challenge, and we got our paper accepted for it, the lovely journal. So that sort of started my inquiry, around reproducibility. I think, we spend a lot of time on just making sure things are well documented and that if somebody gets it, they can build it or run it again, but I think there’s even more complex issues that we talk about less because, they might not be a priority because we’re just working on the basics right now but I think there’s also interesting issues like, at what point is reproducibility acceptable, or when is transparency acceptable?

Because there are some things, especially in the HPC world where it’s just not feasible or for whatever reason, so I feel like it becomes a philosophical issue in a sense, so after being a research software engineer for several years, I went back to get my PhD in history and philosophy of science to sort of explore these issues. And I’m interested in researching standards related to reproducibility. So that’s sort of my motivation for pulling this together.

Bilel: There is also important important fact that over the years how submission has improved, as like the team that has, taken care of the artifact description and evaluation. We make life for reviewer. Because it’s a lot of work. And so the, this year, like they made like almost like compulsory to have at least a container.

So that for reviewing is at least we can check and validate some early result. And some of the reviewer would not be able to have access to some exotic software or uh hardware. So, the container will be able to reproduce it from your laptop up to if they have access to a supercomputer. And of course in a site, if they have a well-documented, source or easy access, how they can download it. Because now the HPC world is like a village. And there are some people who do not have access to these privilege. Uh, I would say resources. So with this transparent initiative, so the student, anywhere in the world, they read the paper that interested they can get the data, because sometimes we think we focus on the code, but also in the productibility they make sure that the data, because it’s not just to do like scientific things, it’s not about like hpc, like what we see here at the conference. Many track are towards science because you just don’t do computation for doing other things. There is something, a goal behind what is the problem you to solve. So many, I would say in biology or in climate. So you need the data. So how you will reproduce some result, because it’s not, the reproducibility is not only about performance, but also some result.

Rocio: I just say that it’s not about open source code, but open science that involves, everything. It’s a collection of code data. The logs, the way you can execute the requirements, maybe an adjustment to your code so it can be used in a smaller environment. So it’s the full collection of everything. Yes.

Bilel: These two they help a lot and they went through the journey of starting as a reviewer and up to being chair, and that’s why I selected them. But no, I’m just stating so that you can tell them their journey. So it’ll be nice to hear how they, transition for being like a reviewer and up leading the effort.

And I think for the last three years we have been there.

Le Mai: Yeah. For the reproducibility initiative, I started off with the repro challenge in 2019. Just helping out with reviewing there. And then in also in 2020 with reviewing and grading, and also helping out a little bit with a special issue in 2020.

2021, I chaired the repro challenge, so I got to actually have my hands on it and also work with the initiative. When Carlos was the was chair and he was the one who really pushed for, a lot of the ADA initiatives and bringing the award to light and. Anyo and Tanu, I can’t remember their last names, but they really, created a framework that Rosa and Roci have built on of, finding reviewers who will evaluate these artifacts and these artifact evaluations.

Finding people who will give, access to exceed machines or access machines at this point, or other kinds of, compute time so that they can evaluate these things. It has been in the last, last five years, and the fact that the repo challenge has still become, a flagship application in, the, the challenge has been really, incredible to see.

I think especially with the award and it actually being called an award for behind, SC has really been amazing. And, yeah, as Bell said before, and as I’ve said, the Ada e like it is a, a lot of legwork in terms of what they do, it’s really impressive to see.

Bilel: So when we look at the 2015, there was like one paper that voluntary, like shared like the artifacts. And, for SC 22, we have over 37 paper awarded with full budgets. So this means like over 45% of paper with three budgets, which is the best thing. So this is already a record, so every year people are submitting and this time when understand it takes time like to do, to collect the data to, put like a, in a proper manner into container.

And what we did this year, and thanks to the previous. Before the SC submission, couple of months ago, we did, some training and we had like this , seminar because part of the reproducibility is also education. So how you can do it, because people, when they look at the container, they’re scared, okay, I can’t do it.

And so what we did, we did three seminars, in the month of March, and they’re all available on YouTube in the main site of SC 22. And one was about like some tools like ecp, like how they can get this E for S. So this is how to, you can get access easy to many software built so you don’t need to reinvent and how to figure out how I can install some libraries.

Then we did another one with chameleon team, so how to get access to some. Exotic hardware from basic CPU to the latest ARM, like the similar one, like at, the Fuku. And last but not least, we had also a similar with the Jet Stream two people, so people can have access to larger resources. So that was like make also not only the life for the participant, the author for submitting paper, but also the reviewer. So the seminar was very beneficial to everyone. .

Nicole: Yeah. So that brings up a great point, to go also back to the award. I think, education is a really important part of making sure that people are able to do these things. But I think also, we have to incentivize people.

So whether it’s the badging or the awards, these things all have to go together before they. Before you reach your 40% with three badges, which is awesome. I love that. So what do you see in the future? Like what are, what is the next step? What is the next piece of education or incentive or the next piece that, continues this trend?

Rocio: I think it’s gonna be a mix of providing good guidelines accessible to everyone so everyone knows how to proceed either they want to reproduce or they want to make a reproducible, contribution. And I also think it’s a matter of all of us, kind of force everyone to made the results accessible and their codes accessible because it happens that if, unless there are legal issues that put a barrier, of course, unless that’s the case, why when you submit a paper, you don’t make the code available so the reviewers need to trust your results. I mean, we live in an environment, an HPC environment that provides you with all the necessary tools to share your code, right? So it’s a matter of education. And also, from the reviewers perspective of the journals, the conferences a model of taking responsibility and guiding everything through the mandatory of reproducible initiative, in my opinion.

Bilel: So today for the group disability, I see one main challenge is the standards of the badges. Today we have multiple badges award. Oh, so you have the acm, which is the one of the most major and most used, but you have also the IEEE, you have the open source. So which one? And each one has different criteria, and I think like the community, so we should ask some of our, predecessor or into libraries like what they did for MPI.

So if you, like, Jack said like when they had the pvm, they had so many, different libraries for the distributed communication. Then they come up together and they said, we do the MPI standards. So I think like this is the next effort that the community, the leadership come together with the editorial from ACM, IEEE, one family and toward a goal, how to make the standard.

And so it’ll make easier the life for the author to submit something and also for the reviewing and the people who try to reproduce the result and hopefully how, also in the long term, how we can make sure that this data stay forever.

Le Mai: I think that Rocio and Beel hit a lot of the same things that I was going to say. One thing I’d like to add about the reproducibility initiative in SC proper is what they’ve described. One, they’ve been coming from the leadership down in terms of, "Hey, if you wanna submit something, you have to provide this and we have badges and we will recognize you if you do this."

So that’s coming that from there, and for the papers. So already there’s that framework. And also that we will help you come to this by creating a framework where we will be evaluating this. And then the other part that I thought was really interesting was padding it from below, which comes with a student cluster competition.

By knowing that we have these, students who will, like you, come into work in HPC or in computing or in the sciences. And with this, they not only have experience, in the professional sense, but also because they’re doing the reproducibility challenge. They’re already thinking why is reproducibility important.

You know, what is good science? What does open science mean? And so there’s that thing where we’re coming up from who’s going to be filling in our shoes when we are going to be retiring, and also from, the people who are currently doing it, and also the top level, making sure we recognize the people who are pushing this forward.

Nicole: Yeah, that’s all really great. I heard a couple things in there. I think that there’s both right? Top down and bottom up . And I think you had also mentioned like a kind of a culture change in how we are incentivizing people to get this done.

You had also mentioned how we talk about reproducibility, right? We have these three different badges. We have at least three different terms by which we speak of reproducibility. But there’s also all the standards science wide.

Uh, and I feel personally like there is some disconnect between the way that we talked about reproducibility, how people outside computation talk about reproducibility, ACM switched theirs, because there’s different communities using different things. And so I wonder if you have any thoughts about the language surrounding reproducibility and what needs to happen there so We’re more productive, not just in our HPC world, but also when we come together and talk with like broader science, how we’re having more productive communications there as well.

Rocio: Well, uh, Jack was saying yesterday that when they had in mind MPI what they did was meeting, if I’m remembering correctly, each six weeks for three days. So they fully focused on that. During that time they said, those things need to be solved, so let’s figure out how to do it.

So I think the best idea would be combined people from different, different levels. So maybe also from students to the highest levels in the hierarchy of, big companies or big conferences, committees. So they put together their thoughts, their needs, their, barriers face.

And it does not depend on a particular organization, but common thoughts put together scientific environment, let’s say. That’s my opinion.

Bilel: Probably one thing, to continue about the challenge that, as you said, opportunity is like kind of more open science, but SC without the industry or vendor cannot be done. So the next challenge is, how the industrial partner or vendor will be able to share things. So there were like, nice paper, but unfortunately they were not able like to share full details about, some codes, which we understand because this is a business, but there should be like a common ground, or this is probably the award, like how to standardize.

Le Mai: It’s really hard question to answer. And I guess, I think Rocio touched on this, which is bringing scientists and computational scientists and as you mentioned vendors together to understand the importance of what they’re doing and how if you’re going to simulate or use your HPC to do science or research, that will then be peer reviewed about how important that that foundation is and without which we are on shaky ground with our results.

Oh, also in understanding where scientists have to understand that, computation has a lot of barriers in front of it in terms of sharing open science. There’s the industry part, there’s a heterogeneity of the hardware and the software that we’re using, and that maybe does not come across as well in the scientific community. So I think that that maybe discussion might help.

Rocio: I just want to add something that maybe it’s not everything can be reproducible, so maybe we need to learn that there’s open science on the one side and then companies software on the other. I’m not saying there are enemies, but I’m saying they have different target, different group of users and maybe we need to learn that they cannot coexist in the same.

Nicole: Yeah, yeah. So I agree. And I think, there’s also things to be like learned, right? There’s absolutely two different needs there, but also there are maybe things that we can borrow from each other. So what comes to mind for me is, you know, you say, oh, well, industry has things they can’t make available.

Well, there are some science like biosciences or whatever, where like we do have to either anonymize something or take other measure so things are both, you know, reproducible, but protective of bulls privacy or whatever the issue is. So not all science can be open as much as we would love it to be.

I guess unless you have anything else to add, I would transition into what your day jobs are and how you feel reproducibility comes up in them. Okay.

Rocio: So what, I’m an assistant professor there in Spain, in Universidad, and I combine teaching with research in my position. So when it comes to teaching, I’m starting to force myself to make the students put their assignments through git. I create private git repository so I force them to use it and submit everything and try to update it each time they make a change, not only the last day because that means you could have done it in whichever way. So it’s good to see the progress. And then on my research side, I did not realize, but I was lucky in the PhD because all the software I was using were, the libraries I was using were open source.

So let’s say I learned about reproducibility when I needed it. And I did not have it because I was, so used to have it that I missed it when I lost it. And it was when I started, working my research with, uh, fluid dynamic applications because I have read more than 20, 30 papers that claim for certain results following specific paths, but I haven’t been able to find their source code.

I would say like more of 90% of the cases. And in some of them I found them, but it was not usable for different reasons. So that’s what I’m facing right now. I want to compare my software with other existing softwares and it’s being very difficult. So I wish, I can contribute somehow with reproducible, software and I can have other people, do the same or follow the same.

Bilel: So my deal I’m Bidel. I’m a competition scientist at the KAUST Supercomputing lab, and so my daily job is to make sure that our user are happy with the super computer, Shane. So basically to summarize, this is the machine, the HPC system, super computer should work as expected when we purchase it and we accepted it. So it means performance, reliability, functionalities, and accuracy. One thing like what we do, like, time to time, we check that all components are working. So we don’t want the user complaining. We want them all happy. And so time to time we have to check and validate the result. So what does it mean, how we can reproduce the results? So we have of course codes.

We have tests that are, some of them are open source, some are priority, but this is already shared by the team of computation scientists, but also with the csad means and also with the onsite. So that the day that I am in vacation or I’m not here or in conference that don’t wake up me at 2:00 AM okay, Bilel, we need your help. And we have to keep some of historical data. So are we sustaining the same performance? And what will be the criteria of, is it okay or are we still, are we reaching the critical thing? So, because over time some performance made, But sometimes also it may improve with the new software and so on.

So we always think about code availability and so on, but also keeping the data, history covata, and to see how over time it improves.

Le Mai: So at IU, I’m part of the applications team, so quite a bit of my job as well as keeping users happy and, benchmarking machines for acceptance that we have.

And, I’ll echo what Bilel says in terms of benchmarking, and performance evaluation. That’s something that, one we wanna see, like, let’s say something happens in the hardware later on or something. A user is coming in and saying, oh, sudden I’m seeing performance degradation. It’s so helpful to come, go back, let’s say to your old HPL runs before the acceptance.

See your configuration file, see what you got then, and be able to, oh, I have that config file. I’m gonna compile it again. Or use that old binary that you compiled before and be able to test that. Another thing was, when I first started, my job in with the applications team, a lot of what I was doing was, doing analytics on data coming in from the systems in terms of usage, et cetera.

And for that, that was like a team-based thing. So where something where if I’m writing the scripts that is pulling the data and and doing analytics, if I move into another job, how can I make something that somebody can see what I did if I did something wrong and improve on it, pull more data, et cetera.

So I guess in that sense, reproducibility, exists in my work. Another thing I’m thinking of is when working with users at iu, we get a mixed bag and some of our researchers may not have heard of GitHub or are doing something perfectly fine on the systems. And then one day they bring the systems to a screeching halt, and we ask them, what’s changed?

We don’t see any changes in your job scripts. And they’ll say, oh, I changed my code. It’s like, oh, well, can I see any, what version you’re doing here so I can make a comparison? And many times they’ll say, oh no, I made these changes and I’ve changed it already, and I’m not using GitHub and that is an opportunity to talk to them and say, well, it would be really helpful if you did something like this or that we could do this, and we create that partnership.

So yeah. Another thing I thought about was, in terms of institute to institute, sharing, at one point, for example, we were bringing something, a service online that helped, gave a graphical interface to one of our, machines. And we were implementing something to do the load balancing on that for when users came on, since it was a shared resource. Later on our dear rivals. Um, I think you went to Purdue, is that correct? Yes. So our dear rivals, Purdue, were also planning on bringing some things, I think with the same product online that was similar and they wanted to hear about, what we were doing in the load balancing, and we were happy to share it and I believe they’ve implemented it as well on their side.

So in that sense, IU Purdue, you know, sure we’ll fight it out on the field, but when it comes to science, it’s really important to be able to share what you’re doing.

Nicole: Yeah. So I feel that, this is also an important aspect of developing standards, is that community development, right? Because it is not one size fit all as well, and so keeping those communications going really needs to be in the context of specific groups, which is really difficult right?

Yeah, to wrap up and summarize. I’d like to hear what reproducibility means to you. I think, it typically goes beyond whatever we try to define it as. And I think that’s actually a problematic way to go about defining it is over defining it. I think especially when we talk about this more broadly it means a lot of different things.

Whether it’s standards or some other components. What does reproducibility mean to you? Or, what are you most passionate about going forward in in terms of making reproducibility more accessible.

Le Mai: I think the word that comes to my mind when I think about reproducibility in general is a legacy. Right? You’re creating a legacy for whatever you’re creating. You’re creating something that future generations can build upon and make stronger. So if you have something that is reproducible, you know that the ground, the next person coming into this is not shaky. It can hold water and they will be able to build that next step. And so it is a legacy, it’s a human legacy.

Bilel: For me, reproducibility is, I would try to define it with, two words. Its availability, towards sustainability so that over generation, generation and so on, and basically she summarize it, Le Mai, it’ll be so that will have a legacy. Without this, we’ll not have reproducibility. Um,

Rocio: I would say honesty and honor because, if I had to explain what Albert Einstein used to say, you know nothing about a field unless you are able to explain it to your grandma. So I would tell my grandma, you know, when we are at the school, they ask us to justify every answer, and then we get to publish papers where nothing is justified. So we are missing something.

Bilel: There is a joke, like everyone wants to get results fast and so on. So someone said in his resume, I’m very fast in calculus, so the interview was perplex and said, what do you mean? Yes. Okay, you can test it. Okay. So the interviewer said, what is 53 by 27?

And the new applicant said quickly 128. It’s wrong! Yes, but I told you I’m fast. I did not tell you like I will give you the right answer , so it’s not reproducibility .

Le Mai: Yeah. Oh yeah, please. I just wanted to add something. As I was listening to both Rocio, you and Bilel talking about what they thought in general, which was, it’s equitable.

Having reproducibility makes equitable science being able to leave this, it gives anyone, an idea of how they can do this.

Nicole: Great. Thank you guys so much for coming. I learned a lot. I think that summary of what reproducibility is, really explains, how much of an important human issue this is.

Right? It really is very big and you guys are doing a lot to forward the community. So thank you very much for coming today.

All: Thank you. Thank you for sharing that. Give us okay. Yes, continue. It’s a lot of. Because I know like some people when, I try to bring this committee and say it’s too much work, work.

[00:00:53] After college, she worked as a learning center teacher and led psychoeducational groups in a psychiatric hospital. While co-authoring a chapter about neuroscience methodology with Scott Lilienfeld at Emory, she later conducted research in personality pathology as a fellow in Aiden Wright’s lab at the university of Pittsburgh. At Purdue she worked as an assistant clinician and the lab component of a PhD level statistics course, which dovetailed with her research interests in psychiatric classification and statistical methodology. Her substantive projects have involved investigations of affective and interpersonal processes, narcissism, externalizing symptoms, PTSD, suicidality, cognitive neuroscience and psychosis. Her broader research program revolves around developing mathematical models and computational tools, EG scientific software or application in scientific research broadly and in psychology and neuroscience in particular.

[00:02:11] So a very warm welcome to the show. Elizabeth, it’s very good to have.

[00:02:17] Elizabeth: Thank you for having me.

[00:02:20] Nicole: So, first of all, congratulations on defending your dissertation.

[00:02:26] Elizabeth: Thank you.

[00:02:27] Nicole: And this interview is a bit unique because I have actually worked with Elizabeth in the past on a scientific software project. She was a lead graduate research assistant, and I was on the project as an RSE. But before we get started in all that, I want to know more about your background. I actually had no idea that you had such an interesting path.

[00:02:54] Elizabeth: Charitable way to put it.

[00:02:56] Nicole: No, it is interesting. So why don’t you go ahead and tell us how you got here.

[00:03:01] Elizabeth: Yeah. I suppose I have pretty broad interests. And the funny thing actually was when I was younger you know, 13, 14 , there are a lot of different careers I can pursue that I’d be interested in neuro-linguistics international relations, but the one thing I’ll never be as a quantitative psychologist.

[00:03:20] And then I became a quantitative psychologist and I’m like, well, okay, that’s the one I’m interested in, tons of things in psychology and neuroscience, philosophy of science, methodology, stuff like that. The one thing I’ll never really get into is cognitive psychology. And then that became one of my major focus.

[00:03:38] But yeah, I think it, it kind of started out with a strong interest substantively in psychology. And when I got to college, I was a part of this debate society, not like high school debate more of more of like a literary society or, you know, an intellectual group.

[00:03:55] And I got very interested in philosophy. And so even though I was a molecular biology major at the time focusing on neuroscience I started taking a lot of history courses, philosophy courses, et cetera. And so the second week of my senior year, I looked at the classes I wanted to take and looked at which majors that could fulfill and turned out this like super interdisciplinary track of political science would let me graduate.

[00:04:19] I, I had not taken a political science class until the last year of college, but able to complete the major was very flexible in that way. But after, after that I realized how much I really missed doing science rather than just talking about how science is done and how that affects society. And , what kind of inferences people are making or not making.

[00:04:43] So, I started volunteering in Scott Lilienfeld lab and working in the psychiatric hospitals and I started to kind of those two interests in, or those three interests, I suppose, in neuroscience and philosophy and psychology. Those streams kind of merged together. And I think that the natural output of that was of focus on measurement and assessment and developing actual tools, both, you know, in terms of methodology and math and statistics, as well as software.

[00:05:13] Nicole: Yeah. So this is really interesting. We have similar backgrounds. I’m also was an interdisciplinary major that I desperately just tried to string together a whole bunch of classes that I had taken. And I was also really interested in science and ended up in computing. Just because I saw that, I could be kind of a generalist, right.

[00:05:38] Like I didn’t have to deep dive too hard into any, any one topic.

[00:05:42] Elizabeth: Yeah, it’s fun to be a generalist. You get to do lots of different types of things. It is, it can be a little tough in academia because you know the interdisciplinary stuff is not really well rewarded unless you leverage interdisciplinary methods to study, you know, a very narrow range of issues.

[00:06:02] So that’s definitely been a little bit difficult to navigate, but I’d prefer to kind of focus the research I want and then figure out how to make that work later.

[00:06:12] Nicole: Sure. Yeah. So like I said, I know Elizabeth through a project we were both on, related to power analysis of study design. Elizabeth worked on the backend computational software. And i worked on the graphical user interface. So why don’t you go ahead and tell everyone about what that project is and your role in it.

[00:06:41] Elizabeth: Right. So we are designing software that essentially. Researchers figure out how to design their experiments, in particular, how many people to recruit in order to have the, have a good chance of detecting the effects that they want to do. And so this is required just like a range of kinds of approaches to different statistical models and figuring out how people think about. Their statistical analyses and their experiments.

[00:07:10] But it’s been definitely a very cool project.

[00:07:15] Nicole: Sure. So this is a computationally. Intensive project. Was this your first experience doing computational work?

[00:07:25] Elizabeth: This was my first substantial experience I had, used, R and M plus and stuff. So. Sort of coding, you know, for isolated analyses, but I had not really done very much in the way of kind of a formal software development. So it was a very new experience. And I started on with the, the PIs had just been granted the and in terms of coding the actual software, I was the only person doing that.

[00:07:57] So I kind of started. Alone doing that and testing it and trying to set up some infrastructure at a time where I had, probably a mere months prior learned what version control was, you know, so certainly challenging. And particularly as we started adding more people on the project and kind of learning how work in the context of software development team and interface with the UI UX people and, you know, working on the GUI. I had to learn quickly.

[00:08:29] Nicole: Yeah, absolutely. You self-taught a lot of languages and technologies. And if I can say, so you do a really good job advocating and teaching yourself best practice.

[00:08:43] I’m wondering if you have any lessons learned or any advice for grad students that might be in similar positions? I think it’s a pretty common occurrence, right? For, for grad student to sort of be in charge of all this stuff.

[00:08:59] Elizabeth: Yeah. It’s definitely been an kind of trial by fire. If I could go back I definitely would do things a lot more cleanly. But I think that sometimes you’re just stressed into a project and particularly if you’re the one trying to spearhead it and teach yourself a bunch of things. I mean, I’ll give advice with the caveat of, this is the way that I did it and it worked out okay.

[00:09:23] I guess maybe like one pathway there may be others, but in terms of my perspective, I think that what helped me most was flexibility. So, just because I had done things a certain way and put a lot of effort into that. If it was, causing problems or coming up short or was not flexible to new needs being able to say, okay, yeah, I worked hard on that and that was working for a while, but now we have new needs and not being hesitant to kind of switch streams.

[00:09:55] I mean maybe two or three years into the project, I switched basically everything from R to Python. And, you know, that was, that was a, big switch. But it definitely, I think, led to us being able to accomplish a lot more things. In terms of branching out into new modules, adding new features to the product. And I mean, I don’t know whether you agree or not, but I think, I think it made it a little bit easier to hook up to the user interface.

[00:10:26] And you know, just my version control systems, you know, just being flexible to noticing when things are getting messy, when people are having difficulty like staying on board and Yeah, that that was the thing. And then the other thing is not getting scared off. I think a lot of people when they are thrust into a project, whether it’s in a management role or as you know a member of the team, not in a management role, People get, can get overwhelmed, especially if they’re early career and it can seem like there is nothing you can do.

[00:11:05] Like it’s, it’s hard to know where to start. And, you know, I think that if you just plow ahead and break things down and just figure out one thing you don’t know, and then get on stack overflow, get on Google, get on, you know, YouTube, however, whatever resources you can find. And if you don’t get scared off by the problems that seem overwhelming, you know, as you start to kind of research things and delve into resources and, you know, just try to write code things kind of can unfold a little bit more naturally.

[00:11:39] And I think that a lot of people just get stuck at the phase of this. It’s so overwhelming and there’s so many things I don’t know. So I, I don’t even want to get started or I want to, have someone else do it, or I, will just kind of go to someone else and say, I’m having this problem, which is also good.

[00:12:00] It’s good to reach out and collaborate with people. But, you know, I think that being willing to just open that tab, go to Google and just start exploring is, is a major step.

[00:12:13] Yeah, I think it is definitely a skill to learn to just try it and get information from trial and error just by planning or researching.

[00:12:26] I think that’s a really great point. You brought up that you were doing a lot of training and documentation. You were doing a lot of things in a managerial position. So what do you feel like that you’ve learned about onboarding students and particularly people with non-technical backgrounds?

[00:12:54] Yeah, I think that one thing I learned is that I’m good at training. I’m going to teaching I’m good at documentation and I’m not a great manager. You know, I, the day-to-day stuff is a little bit more difficult for me, keeping on top of, early on handholding and really making sure they understand what’s being communicated, but hopefully I’m getting better.

[00:13:19] I think that. It’s very important to orient people to where resources are. So if you have, you know, training documents or like orientation materials and things like that consolidating that as much as possible is very helpful. Because a lot of people get tripped up because they don’t know something and they know that there are resources that have been developed for training, but, you know, they don’t know where.

[00:13:45] And so having a document of this is where this is, this is where that is. And having a few of those kinds of repositories of information as possible is very helpful. And I found ways actually to integrate these kinds of materials into the repository itself. So, into the discussions tabs, depending on what version of get hub you’re using an issues tab and having step by step.

[00:14:07] Instructions on how to do things, how to test code and. Basically what the point of testing is that that can be helpful because then things are all in one place and people can discuss and ask questions among themselves and posts like, here’s this weird issue that I had on my machine. And, here was how I solved it.

[00:14:28] And then some person will randomly have that issue because you have like different drivers or different, randomly. Silly packages one version too old or something, but that’s mainly what I’ve learned so much. It’s been kind of, kind of a trip.

[00:14:48] Nicole: Yeah. Well, I feel like this role that you had had quite a bit of. Responsibility. And I wonder, what steps that you have taken to hopefully mitigate some of the inevitable burdens that the transfer of power in a big long-term project, like this.

[00:15:13] Elizabeth: Documentation documentation is just so important and creating documentation with a mind that someone else is going to be reading it. So you don’t want to assume any information. Right. And so I have. Very long docstrings for most of the major classes and functions and vignettes explain exactly how to use of each feature with sort of narrative explanations where appropriate.

[00:15:43] And I think that hopefully that’s going to be helpful for people. I particularly on a project where we’re going to be recruiting. A lot of people don’t have a ton of experience in software development because we tend to recruit research scientists who know about methodology and statistical analysis and experimental design.

[00:16:01] And, you know, they have some coding skills, but maybe not used to a project of this scale. So with that in mind, I would say probably a good 70% of my lines of code are common. With very explicit, this is what this stuff. And definitely also training people on how to use the debugger not just for debugging, but for understanding code, you know, basically opening the hood and seeing line by line exactly how the objects are manipulated.

[00:16:29] Exactly. What’s going on. You can really help them understand the logic.

[00:16:36] Nicole: Yeah. All that’s really great. And I know part of the, this training you were doing is getting people hooked up to the campus clusters. I want to know how HPC was related to this project.

[00:16:52] Elizabeth: Hm. So a lot of what we’re doing is simulation-based, so we’re doing the same thing over and over and over again.

[00:17:00] And a lot of these things are very computationally intensive, right. You know, when a researcher is doing it, they only have to run that statistical analysis once and they already have the data. We have to simulate the data and analyze it. And even if say that analysis would take two minutes for the research.

[00:17:20] You do that a thousand times. And you know, you had quite a few hours of work and, you know, eventually we started to try to implement parallel processing and things like that, but that can also get tricky depending on the packages that you have to use for analysis. In addition to that we had to run a lot of different tests because.

[00:17:45] I think unlike a lot of software development projects where you look at it and if it runs without error, you know, it works, you know, more or less, you know, if you can just see that it works. For us, we need to also verify that, it’s working exactly in the way that we intend under the hood and that the math is right.

[00:18:05] And sometimes it’s difficult to establish very clear external metrics by which we can judge our output. And so, I developed testing systems that would basically go through all the combinatorics every way the user could specify something and try to compare those to whatever either a Priore or empirical metrics that I could develop to make sure that we’re actually simulating what we wanted to simulate, that we were analyzing it properly. And this requires a lot of tests, a lot of tests with a lot of both numeric and combinatoric coverage. And so it got to the point where this is just not something that could be run on a local machine.

[00:18:49] And last we wanted to wait weeks and weeks and weeks. So we got the supercomputing cluster and that’s also been quite a trip learning how to do that, but it’s definitely sped up the process substantially.

[00:19:01] Nicole: Yeah. So I think this is a really common problem. Having to validate computational code, make sure that, you know, you’re numerically getting something reasonable.

[00:19:13] So what is it that you’re looking for or that you’re trying to protect against when you do this kind of testing?

[00:19:21] Elizabeth: Mainly me doing misunderstanding the math, underlying the statistics, you know, it’s like, oh, well, I thought that was a simple matrix multiplication, but actually that’s a chronic or product, you know, or or just that I, in some line of code one bug I found recently is I.

[00:19:40] Forgot to add one index for a data frame that I didn’t add. So, it ran without error and a lot of the numbers look reasonable. But some of them did it and it was because I was not adding that column of the matrix into that, into that multiplication. And so, you know, such a simple thing, if you characters and you know, it, completely changes the way that things are calculated. And so mainly I’m trying to protect against my own inexperience and, or confusion.

[00:20:13] Nicole: Sure. And I don’t even think it’s necessarily experience that prevents us from making bugs. I think that’s just a, sort of a certainty that comes along shore coding.

[00:20:25] So I think that’s a Valiant effort often not taken

[00:20:32] . I appreciate

[00:20:33] Elizabeth: that. For me, a lot of the statistical concepts that we’re trying to implement, in fact, I would say possibly even a majority of them. I didn’t really know when we started. And so I, that was another thing that was just hours of Googling and reading papers and trying to make sure that I understood how these statistical analysis actually worked and , when you teach yourself something, a lot of the times it happens imperfectly and so that, that definitely has been quite challenging.

[00:21:11] Nicole: Yeah. All that’s really great. So now Elizabeth, I would like to know about your own research. You did, you just defended. So why don’t you tell us what you were working on?

[00:21:22] Sure. So essentially when I got back a little bit to my cognitive neuroscience roots and applied it to clinical psychology and what I was trying to figure out if there are ways that we can use mathematical modeling to identify people who have schizophrenia And the data that I had, it’s an archival data set.

[00:21:43] Been in the works since I was maybe three years old. So I don’t, I don’t claim this marvelous data collection, but, I was lucky enough to inherit it. And it was a simple cognitive task that involved basically speed accuracy, trade offs.

[00:21:57] So, you try to go as quickly as you can while still being accurate on. And we know that people with schizophrenia tend to have more difficulty with this task. They also tend to have more difficulty recognizing and adjusting when they make a mistake. And so for most neuro-typical people on this very simple task they might answer a little bit too quickly and say, oh shoot, you know, that, that wasn’t right.

[00:22:23] And they will slow down. And I was interested in modeling the within person process. So they do multiple trials of this task and I wanted to see, can I characterize using just a mathematical formula an individual person’s process of converging on that optimal response time and responding and altering that process if they’ve made a mistake. And so I applied a variety of mathematical models, including some piecewise linear and exponential models and some damped oscillator models and just ran them through some machine learning classification procedures to see if I could, use those model parameters that I fit to each person to identify people with a schizophrenia diagnosis.

[00:23:09] And you know, my best models ended up with 81% accuracy, which, in this area is pretty good. So I think that pending replication and other samples and some refinement, this could be something that is promising for clinical implementation.

[00:23:25] Excellent. Yeah. That’s that’s interesting stuff. What is your big next step now that you’ve you have all this research and computing experience under your belt?

[00:23:37] What what’s next for you?

[00:23:39] Elizabeth: Oh, boy, that’s a big question. So I’m, still kind of figuring out my next steps, but I have a potential kind of soft offer for post-doc position at the Yale school of medicine. Funny, funnily enough, I’m ending up in the same exact building where I conducted my undergrad cognitive neuroscience research 11 years ago.

[00:24:03] And so yeah, it’s, it would be, it would be very interesting. And there are two different startups that I’m somewhat involved in. One of which I co-founded the other, which I’m just in this past week have started getting involved with talking with the founder who coincidentally enough.

[00:24:20] She’s a friend from college is working with one of the former mentors of my potential post-doc advisor. So small world, small world. But essentially what I want to do research. And there are many avenues for that. Postdocs, there are military labs, there are national labs. I expect that one day I’ll end up in national lab or military lab at some point in my career.

[00:24:43] Yeah.

[00:24:44] Nicole: Yeah. And so not that you have to know now, but when you’re considering all these career paths, What are the most important factors for you right now that are in the

[00:24:56] Elizabeth: balance? Yeah, so I want to be able to do interesting research that will have high impact.

[00:25:05] I want academic freedom, you know, I want to be able to, choose my own projects. I’ve been. Uncommonly fortunate. And my grad school career to have an advisor who lets me do whatever I want. So I definitely don’t want to, now that I have my PhD downgrade and have less freedom than I had as a first year grad student, it’s a little hard to go back right. And so, you know, definitely, you know, while some industry positions are interesting , I’m not going to be able to choose my own projects. And so choosing my own projects, I think is probably the biggest consideration.

[00:25:43] Nicole: I definitely understand the draw of intellectual freedom.

[00:25:50] So what is it now that you’ve joined USR? What is it that you hope to get out of your membership to that community?

[00:25:59] Elizabeth: You know, I think that we increasing collaboration between more kind of theoretical and basic researchers and actual software engineers is super important because a lot of projects in academia get published and they say, future directions, here are some applications, no one ever applies because it isn’t rewarded. And the training and how to implement these things is uncommon among researchers. And so I think that, you know, increasing communication between people in university contexts and people who actually know how to create the infrastructure to support the dissemination of our scientific findings is really the next step in actually making fields like psychology and neuroscience have a broader impact in one, in an impact that’s more immediately. Rather than just sort of trickling down across decades.

[00:27:01] Yeah. I think very eloquently put, I have nothing to add. So I want to thank you very much for, for coming on. And also I’m looking forward to hearing where you end up. I’m sure you’ll have just. as interesting and non-linear of a career as where you started with your education and then I’m sure it will also be very interesting.

[00:27:31] Oh, I, I certainly hope so.

[00:27:34] Nicole: So thank you very much.

[00:27:36] Elizabeth: And thank you for having me.

[00:27:39]

Nicole: Hello, and welcome to the long tails of science podcast, where we interview women in high performance computing about science research, mentors, and career paths. I’m your host, Nicole Brewer. In this episode, we talk with Nancy Wilkins-Dier about her pathway into HPC. We discussed her lengthy career at the San Diego supercomputer center. Her involvement with Terra grid exceed and science gateways. And we also discussed the state of software related contributions in academia. Finally, we end with Nancy giving a little career advice to students and we catch up with her on her recent retirement.

[00:01:08] Introduction

Nicole: Nancy as a former associate director of the San Diego super computing center. And she has been there since 1993. And she’s now retired. She’s held a variety of management positions and her particular expertise is in the development of web interfaces to high-performance computing systems, data, collections, instruments, and other resources, fundamental to many of today’s research endeavors . Beginning with the NSF’s TeraGrid science gateways program in 2005, she has led many programs that support the democratization of access to high-end resources. Most recently, she has served as co-PI on both the NSF XSEDE program and the Science Gateways Community Institute and NSF scientific software innovation institute. She has held a number of leadership roles in NSF projects funded over hundreds of millions of dollars over her career. Nancy received her bachelor’s degree from Boston college in mathematics and philosophy and her master’s degree in aerospace engineering from San Diego state university. Welcome to the show, Nancy, and thank you so much for joining me.

Nancy: Thank you very much, Nicole. Thanks for having me.

Nicole: So I’d like to get started with just getting to know you a little better. What do you like to do in your free time?

Nancy: As most of my work colleagues know I do a lot of running even before I was retired. I was well known for that. I used to organize runs at a lot of the major conferences like supercomputing and PEARC, and basically anywhere I went to a conference, I was organizing some kind of run. So I got to meet a lot of people from a lot of different institutions and have some really great experiences out on the trails in salt lake city or through new Orleans bourbon street in the early morning hours. So it was a source of meeting, lots of great colleagues, some of whom I still keep in touch with. That’s pretty cool. I actually went to PEARC and I’m sure that you were the one that organized it in Pittsburgh, there was a run. Yup.

Nicole: That’s awesome.

[00:03:22] Pathway to Compting

Nicole: Let’s start with your initial experience in technology and computing in general, because I think it can be interesting to hear the perspective of women because the story doesn’t always start with, I was five years old and I built a computer.

Nancy: No, not at all.

Nicole: So what was your first exposure to computing and what got you interested.

Nancy: Yeah, it was kind of a non-traditional role, really. Like you said, I wasn’t building computers at five. I didn’t have this lifelong interest in science. When I graduated from high school about the only thing I knew about what I wanted to do is that I liked math and that came easy. It was fun. So that led to a college degree in math and it wasn’t until the end of the. Four years of undergraduate. When I actually realized that math could be applied to things, we proved a lot of theorems. There was not practical applications of math and the program that I was in. So we finally had this professor from bell labs senior year, and then I realized it could be applied to engineering. And that was like a revelation to me. I was very sheltered.

Then I decided to go for a graduate degree in engineering. And when I was first starting at San Diego state, the boyfriend of one of my college friends who was at Stanford, also in an aerospace program, and he was working with Robert McCormack, who is one of the early writers of computational fluid dynamics codes. This was in the eighties. So this stuff was just getting started then there wasn’t really a whole lot of computing going on back then, but I was fascinated by this and the fact that you could model physical processes with math and then programming it into something that computer could understand, visualize the results I thought was really cool. That was way before the great viz that we have today, we had some pretty simple stuff back then, and I was still impressed, but that’s how I got into it. I was fortunate to have professors who kind of let me do my own thing at San Diego state, cause nobody was really doing that. So I was able to pave my own way. I use the academic computing systems at night when they weren’t being used to process student records.

So I had this account on a VAX that I could run on all night long, but not during the day. And I had a few precious of Cray time at the San Diego Supercomputer Center that I would say for like really special occasions. I mean, I had like an hour time, total and a quarter or something, which is like, just like laughable now compared to what people had. But we had these block grant programs that went to universities, I think in the beginning it was throughout California. And now of course it’s everywhere and there’s the campus champions program and all these great things. But it was kind of funny because I was working out of a basement lab where I could have 24 hour access to a terminal and we couldn’t make phone calls from the phones there. You could only call within San Diego State could just dial extensions. And so anytime I had a question, I had to run up to the payphone in the courtyard at the engineering building, put it in my little quarter and ask a question. So one of the first things that I didwhen I finally got to work at the supercomputer center was installed an 800 number. So other people didn’t have to do what I did to get their questions answered. So it was kind of roundabout, but…

Nicole: I just finished Outliers actually and the book talks about how, when you think of all the tech people that started all these companies, a huge part of their success is always how many computing hours that they can get in on these machines back in the day where there’s so many physical limitations, like being in the right room and being able to make a phone call. So that’s super enlightening and very cool.

[00:07:12] Transition to San Diego Supercomputer Center

Nicole: So then from there, how did you start getting involved in this sort of intersection between software and engineering and science?

Nancy: Yeah. So my first couple of jobs out of grad school were at engineering firms here in San Diego. So I ran computational fluid dynamics. CFD is the acronym I’ll use cause it’s easier. CFD simulations for missile design at General Dynamics, which was a really large employer at the time I graduated. And then moved on to General Atomics where I was doing designs for gas, cooled nuclear reactors, so helium flow through a reactor instead of airflow over a missile, but similar kinds of concerns and codes and things like that. And both of those programs ended, there was like a shutdown in funding for the missile systems at GD and the same thing for the reactor program at GA. And at the time general Atomics was running the San Diego supercomputer center, which sits like across the street, on the campus of UCLA. So it was an internal transfer, just looking for another position and ended up getting the position at SDSC and then staying there for like 25 years.

So there was so fascinating and I liked it so much more than engineering, especially in a large company, you can get really pigeonholed where you’ll do the same thing over and over in slightly different ways as they make a little design changes. And with supercomputing, people were using the supercomputers in all, all areas of science, not just engineering, chemistry, biology, and now it’s even social science is just so thrilling. And so there I was never bored. I had a position answering the phone when I first started the help desk, the same number that I called as a student. So that was kinda cool, but just talking to people from all over the country, trying to do all sorts of different things and then having new technology and new computers to contend with really every couple of years, I mean, it’s easy to have a career like that.

Nicole: Well, it’s a trade off, right? It’s sometimes in the academic side, you know, we don’t get paid quite as much, but there’s people here for a reason. There’s something intrinsically interesting about being in this intersection and it’s really dynamic, I think.

Nancy: Yeah. I agree. Yeah. Yeah. Really rewarding too.

[00:09:34] TeraGrid

Nicole: So a lot of your work is focused on software that makes computing systems more accessible and maybe TeraGrid is the first installment of that.

Nancy: Yeah. I mean, you can’t always plan out a career and how it’s going to go and what twists and turns that are going to be there. And that science gateway program really changed my life, changed my career path. And Charlie Catlett, who I think is still at Argonne National Lab to thank for putting me in charge of something like that. So that was kind of early two thousands and the web was just getting going. And prior to that, people accessing super computers. You didn’t have to be at the facility of course, but it was very much interfacing at the command line and working very directly with the supercomputers. So we’re starting to see, the web develop and scientists using the web. You have these large communities of people who need this processing power, but don’t have all the background training in HPC or all the expertise. And so that was kind of the genisis of that program was looking at some of the large NSF projects that were web based and how we could work in a very hands-on way with eight projects to what did an interface to a supercomputer have to look like so that you could go in from the web and they, I want to do this simulation. I’m going to construct this Fortran code and I’m going to submit this code through a web interface to the computer, but even stepping back from that, like, I just want to solve this scientific problem. And describe it in terms that I know how to describe. And then on the back end, it’s going to construct the code and launch the code and then deliver me the result. And it was a very new way of accessing super computers.

We had a large working group, including security professionals. We had to make policy changes with the national science foundation for how these things can be accessed, because you can imagine there’s like a lot more anonymity and a lot more trust that has to be there to use these expensive resources, but we were able to navigate all those roadblocks and create a very successful program that still operates today. Now there’s hundreds and hundreds of gateways that use the supercomputers that the NSF funds. Yeah. I’m really proud of that, but it was all just sort of, you know, luck. Cause I think I was involved in user services still with like the consulting aspect of supercomputing and him just saying, “Hey, how’d you like to do this?”. Okay.

[00:12:03] Transitioning to XSEDE

Nicole: You’ve been involved with both TeraGrid and XSEDE. So maybe we step back and talk about what the goals of TeraGrid were and then maybe how that’s changed now that we’ve transitioned to XSEDE.

Nancy: So back then grid computing was more of a thing. Now it’s more cloud computing that we hear about, but the very initial idea was to construct a set of distributed machines that were identical, but geographically located apart from one another, and that people would run large jobs across all of these machines at assembled. So there was a great emphasis on the networking.

But really what TeraGrid did was bringing together these different supercomputer centers in non-competitive way. So working together on a single project and so XSEDE has carried that idea forward through John town’s leadership. And I think it’s been a huge win for the centers. Back when I first started at SDSC, we were very siloed. SDSC that Pittsburgh supercomputer center NCSA. Don’t talk to the enemy, man, no working with them. So TeraGrid was the first instance of a project. We were all funded to do the same thing together, and there were some steps forward and back, and I’m not going to say it was the smoothest transition from whatever years of working, one way to working another, but especially with John’s leadership through the XSEDE program and all the great people that work on it, we’ve made huge strides. Had lunch recently with Ralph Roskies, who is the director of Pittsburgh supercomputer center and his wife. And we get together regularly, even though we’re both retired now. So there’s been some huge wins through this programs.

[00:13:43] Science Gateways

Nicole: Wonderful. Yeah, so I’m told that you coined the term science gateways.

Nancy: You know, I wouldn’t say coined. We can’t remember who coined it. I would say I popularized.

Nicole: So for those people that don’t know what is the science gateway?

Nancy: I think of it as a web interface to some kind of capability, whether it’s data collections or. Analysis or remote instruments we’ve worked with telescopes and things like that. The gateways that I’ve always worked with have had that computing aspect. And so we coined the phrase in connection with computing, but the democratization, which I think was in the bio, that’s a word that we like to use a lot because there are an awful lot of groups that don’t have. Two large data stores and high-end computing and things like that. And if you can get access to a gateway and a lot of them are really pretty open and provide access to anyone anywhere in the world, you can get all these capabilities at your fingertips. It’s amazing. Some of the statistics from some of the gateways where they’ve just got worldwide users, some of them from not very rich countries, but still doing very important research. And they’re able to do that because of this gateway where otherwise, you know, no way would they have access to those sorts of things.

Nicole: Are there any particular gateways that you’ve worked on that you’re particularly proud of.

Nancy: The Cypress science gateway, which uses a lot of resources at the San Diego supercomputer center. That’s been a leader for a long time and they do phylogenetic tree analysis, which is useful in a lot of different fields. Nano hub is a very successful science gateway. It’s kind of interesting how some of these gateways have changed. A lot of things too. Cause some of them, it’s just individuals running jobs, running analysis, and not really interacting with other individuals, but some of the gateways really form these communities where people are answering questions for one another and contributing content like code or presentations or classroom materials, all kinds of different things.

[00:15:56] Giving Credit to Software-Related Contributions

Nancy: And in nano hub, for example, they’ve actually been able to change. The tenure process a little bit. So looking, not just at publications and citations, but looking at what someone has contributed to this gateway and how many of the thousands of people on this gateway have found those contributions useful and making that also a component. So I think it’s kind of modernizing a little bit, or what’s considered as a valuable contribution to science.

Nicole: Yeah. And there’s a lot of talk about how do we do that formally to make sure that people are sort of recognizing.

Nancy: Yep.

Nicole: Do you have any ideas on how that could be improved?

Nancy: We have tried to work with people like AltMetrics was a good group that I remember working with, but there were some people doing some really leading work, especially software contributions. Dan Katz had a lot to do with recognition of software contributions back when he was an director at the NSF, which were really important because you think of the value in creating software, that’s heavily used. And there was really no mechanism, especially for open source to kind of recognize how much something was getting used and how you were able to site software and things like that. So he’s done a lot of that of work in that regard. We’ve tried to do some work as far as citing the gateways because they have to secure their own funding. So they have to show their utility to their users. We were always trying to, to work with them in that regard to make sure that they were capturing success and telling those stories in the right way, because it was vital to their survival to.

[00:17:35] Project Managment

Nancy: I just wanted to talk more about your leadership positions and how management is also an important role. And we get divided up into people that want to work on tech and people that want to go into management and maybe why it was the choice for you. Yeah, I think I was good at it. I think it’s kind of interesting working with a lot of tech folks. A lot of people like to be more heads down programming instead of lots of communication with lots of people. And I think it’s great that we have so many different types of people like that because we couldn’t, you know, absolutely couldn’t achieve what we do if everyone were the same way. Right. So I think it’s a good fit for me. And it has been from early. It helps desk stuff for a few years at SDSC, but really within the first five years, I was starting to do management of different programs like back to impact E gosh, in the nineties. But project management, working with large teams. That’s been something that I’ve been good at, and it’s kind of been rewarding, especially in XSEDE’s ECSS program, where we had a large staff of people across close to 30 FTEs, maybe 70 individuals across. ten-ish sites across the country. And what we’re doing is matchmaking requests that come in for a supercomputing support. Could be gateways, could be optimization, could be like development of community codes, all sorts of different things or development of training and matchmaking, our staff to these projects. And then to be able to see them go off together and do great things. And, you know, you know, following up all the interviews at the end of these projects, how did things go? You know, is there anything we can improve and just kind of working on a program like that? It, something that, that I think I was, I was well-suited to, and it was very rewarding because you felt like you got to do more. By allowing all these other people to do more, if that makes sense.

Nicole: Yeah, it sounds like communication and looking at things from the high level are really important.

Nancy: There’s need for all of that in technology, there’s definitely need for the depth of knowledge in particular programming techniques and things like that. But there, there is also the need to step back a level a level, and be able to explain things to the community, explain things to review panels, you know, write compelling proposals, all those sorts of things. So a lot of different skills.

[00:20:14] Retirement

Nicole: What are you most proud of that you’ve worked on?

Nancy: I think changing the fundamental access to supercomputers the NSFs recognition that the gateways were such importance, that they funded a software Institute around gateways and their success and helping, you know, all kinds of different gateways to be successful. That I think I’m most proud of. That was my final, big award before I retired and just work with a tremendous group of people on that project. So having something that you’ve created that’ll live on without you, and sort of developing enough good people to just kind of carry that, what I consider important work, forward. Very satisfying, very fortunate to have been able to do that in a career.

Nicole: Absolutely. And congratulations on that, and congratulations on retiring.

Nancy: Yeah. Yeah. My husband predated me in retirement by some eight years. So he was anxiously after me to join him. And it’s been a lot of fun. I’ve been able to pursue a lot of personal interests. Like I, I took a math class at Stanford through Coursera because I hadn’t done math in a while and I missed that. You know, trying to improve my French, learn the oboe. Maybe I’ll go back to doing some programming. Python’s kind of on my list and never really got a chance to learn that. So. That stuff’s fun. That’s amazing. I mean, have you always just been working even when you’re done with work for the day? I mean, no, and that’s kind of the fun of retirement because I mean, I did work pretty hard when I was working on it, so I was pretty exhausted whenever I wasn’t working. So it’s nice to have the energy and the time to know, take off some of those enjoyable things, you know, now.

[00:21:49] Advise for Students

Nancy: Do you have any last thoughts or advice for the younger people listening? I gave a talk at my high school a while back, and one of the things that I said to those students was to not feel like you had to have everything figured out for your life, that it’s okay to kind of take it one step at a time.

I mean, as an undergraduate, I hated programming because you couldn’t wait till the last minute to do your homework. Cause it always required a little more time to get things, to compile and run. So I just, I didn’t like it. I, if you would have said, you’re going to be working at a supercomputer center for most of your entire career. I said, no, I’m not. So just be curious, be open to new ideas. Don’t I think there’s pressure now to feel like you have to have everything figured out. So don’t, bring that pressure on yourself and it’s okay to just like the next step. I’m going to explore math and next step, I’m going to explore engineering and for you, it might be totally different steps, but you know, don’t panic about that, I guess I would say. And technology is a great career for women. It’s lots of fun!

Nicole: Thank you so much for coming on the show. Nancy, I learned a lot about science gateways and I really appreciate your time. Thanks for coming on.

Nancy: Thank you very much, Nicole. I appreciate being asked.

Nicole: Thank you so much for listening to this episode of long tails of science. This podcast was produced by women in HPC at Purdue, an organization dedicated to promoting and advancing the representation of women in high performance computing. We are a chapter of an international organization by the same name, and you can sign up to be a member at womeninhpc.org. Follow us on Twitter or sign up for our email lists to keep up to date with the new podcast episodes and semi annual virtual meetings. If there’s a guest speaker, including yourself, you would like to nominate. Please send us an email and finally subscribe and rate the podcast on your favorite platform. Or listen on the web at www.breaker.audio/long-tales-of-science

Until next time, I’m your host Nicole Brewer. And it’s been a true pleasure introducing you to amazing women in science, engineering, and technology.