HPC Operator's Review, Sandia Labs, Albuquerque, New Mexico
It's that time of year again for HPCOR; the high performance computing operator's review! This time, my colleagues and I went to Sandia Labs in Albuquerque, New Mexico.
Mon, Sep 9, 2024
This morning I drove to SFO's long-term parking and took the air train to my terminal. It wasn't super crowded, and I got through TSA pretty quickly, although I think the worker using the X-ray had never worked that machine in his life. Our bags were going through it so slowly and at one point there was a logjam of several bags at once coming out of the machine. After I finally retrieved my bags, I headed for the gate, where Norm already was. I had an hour to kill so I played a bit of Pokemon Go, and soon later, Ben showed up.
When it was time to board, it was a bit of a mess since it didn't seem like anyone was following their boarding group number. Anyway, it didn't matter since we all have reserved seats anyway. I got a window seat right above the wing. The flight was super short, and mostly smooth until the end. I read a few chapters of a new book I have called Blink by Malcolm Gladwell, and listened to music.
When we arrived in Albuquerque, Norm got our rental car, a Nissan Altima, and then we headed over to the hotel. Unfortunately, none of our rooms were ready so we checked our bags there and bought lunch at the hotel bar. I ordered chicken enchiladas, which were delicious. After a long wait time to get our food, and after eating, our rooms were still not ready.
So, we left to drive over to Kirtland Air Force Base for gate badging. On the way over, we saw a police operation raiding a house with shattered windows. At KAFB, We saw several other familiar faces there including David from Oak Ridge. It didn't take too long to get our gate badges and then we headed back to the hotel where our rooms were finally ready. I went to mine and it was super nice. I had planned to go for a short walk but after unpacking I needed a break.
The three of us met up a bit later to head off to dinner at Bosque Brewing Co. I ordered mac and cheese and it was really good. I cleaned my plate, but was a bit uncomfortably full after that plus the big lunch we had earlier. I was so tired though, after a quick shower I went right to sleep.
Tue, Sep 10, 2024
I woke up at 6 this morning to get my site badge, which I could use to scan in to enter certain buildings on site. We showed up at the off-site badging office around 7 to see the long line that was already there. When I finally got to the front of the line, it turns out that there was (of course) some missing paperwork or something. It was all on their end though, so I didn't actually have to do anything but wait. The 3 of us from Berkeley and 2 others from other labs all had the same issue. After about an hour and a half spent in that building, we finally received our badges. There was a test scanner that would light up if your badge was active, which was pretty nifty.
After Ben and Norm got their badges too, we drove to Sandia, and used our gate passes from yesterday to get in. Unfortunately, we didn't know where we were going and parked in a completely wrong location. It took me an embarrassingly long time to realize that there were directions in an email we had received (with the agenda) a few days earlier. Finally, we got to the right place after that whole mess.
The conference room was nice, nothing special. There were 3 large projected screens in the front and one on the side of the room. There was also breakfast: muffins, pastries, and cookies. Only sweets, and coffee! I'll need to make sure to grab something in the hotel tomorrow morning. I sat next to one of the guys I met during badging, who seemed too funny and charismatic to be working for the CIA. We started the meeting off with a welcome and introductions, and then each visitor presented their site updates, with Berkeley Lab second, right after Idaho National Lab. I had about 7 slides to present, and I was a bit worried at first, but the presentation went very smoothly. I received a couple good (but not hard) questions, so I was really happy that people were engaged with my topic about data center infrastructure management. We then listened to 2 more site update presentations before lunch at Sandia Lab's cafeteria.
I ordered a ratatoullie sandwich. (What, that's a thing?! I know!) On the menu, it said "cute rat in a hat, sold separately." I asked to buy one but I guess they sold out. (Or didn't have any to begin with.) The sandwich was surprisingly good! I ate with someone I met last year who just started working at Lawrence Livermore National Lab, so we are about at the same point in our career. He was asking me about my experiences with the PE exam.
After lunch, we had the site tour. No phones allowed, thus no photos were allowed either. The Sandia guys made us put our phones in cubby holes before the tour. I'm surprised they didn't ask for other smart devices like Apple watches or AirPods. The tour was really interesting and gave me a view of yet another data center. I probably shouldn't write any details because I don't want the government hunting me down, literally. Sandia Labs is famous for its weapons research and involvement in World War II. So that's one organization you don't want to mess with.
After the site tour, we retrieved our phones and headed back to the conference room. After a few more site updates and discussions, the meeting concluded for the day. My colleagues and I went back to the hotel where I got about an hour to relax before having to head right back out for dinner with everyone. This time, we went to a place called La Salita. There was a long line to order, and some of our group was already done eating before I even got my food. But the burrito I ordered was delicious, and they gave us free sopapillas with honey. I sat next to Nathan from Oak Ridge (we met during my trip to ASHRAE in Chicago) and we chatted throughout the whole dinner. The Sandia host invited us all out to get drinks. I didn't want to go, but was overruled by Ben and Norm, and I decided that I would attempt to be social for a little bit longer.
The bar was nice and we sat outside, luckily the group had thinned out a bit. I didn't order anything but just sat and listened to the others' stories. After a bit, Norm wanted to head out and I seized my opportunity to leave as well, finally getting back to the hotel to relax and recharge.
Wed, Sep 11, 2024
I woke up early to grab the hotel breakfast this morning, but unfortunately it was not included with the room so I decided not to. After Ben, Norm, and I met up we decided to try out a place called Lotaburger for breakfast anyway, which supposedly has good breakfast burritos. I got one with no meat for only $6 and it was in fact really good, and huge!
We had the second part of HPCOR today with more discussions and site updates. Luckily I did my presentation yesterday so I could listen, take notes, and relax today. Although the discussions were interesting, I caught myself losing focus a few times.
Around lunch time, several of us decided to return to the cafeteria, which we couldn't get to because our badges weren't allowing us to pass through a specific gate we needed to cross. So, we decided to go to a food court for lunch and I got a vegetarian Subway sandwich. Keep in mind, all this stuff is right on the Kirtland Air Force Base, along with other stores, pharmacies, and gas stations. You could literally live here, and many people do, since there were full neighborhoods with plenty of houses and parks.
After lunch, we had discussions on power usage effectiveness (PUE) and IT usage effectiveness (ITUE), immersion cooling, and supply chain issues. Soon enough, HPCOR 2024 was wrapped up and we dropped off our badges.
The three of us and someone from LLNL went to the National Museum of Nuclear Science and History on our way out, since it was so close. We saw several fighter jets, missiles, histories of Oppenheimer and Ernest O. Lawrence (who founded LBNL) and replicas of the Fat Man and Little Boy, which were the two bombs dropped on Japan in WWII. It was extremely frustrating to read their histories since they did not do Japan any justice.
The museum closed pretty early so we headed back for the hotel to have a short but much needed break. I took a quick shower and then the three of us left to drive over to Santa Fe for dinner and sightseeing. I loved Santa Fe as it was an extremely iconic and well-decorated town with all its native architecture, turquoise trim, and art stores. For dinner we went to La Fonda, a fancy restaurant where I ordered enchiladas. One of the cooks made guacamole table-side, which was really cool to watch, and was incredibly tasty.
Thu, Sep 12, 2024
There wasn't much time for anything this morning because Norm had an early flight back, so we left for the tiny airport early in the morning. The entire airport was about the size of one gate in SFO. Ben and I decided to stay there and work there until boarding our flight. I planned to buy some souvenirs, but ended up completely forgetting. Then I realized, there were about 3 places to buy food from, and no gift shops. Makes sense now why I forgot.
I ate a breakfast burrito and Ben and I worked on decyphering air handler logic. I also took a short break to walk up and down the length of the airport, which took about 2 minutes.
Finally, it was time to board. I sat down and wouldn't you believe my luck, I had all 3 seats in the row to myself. The flight itself was smooth and landed early which was awesome. I got a bunch more reading done on Blink and saw some canyons from the window in the plane. Not sure if any of them were the Grand Canyon, but it was still really cool to see.