Germany has eased tensions in the Middle East after helping leaders in the region reach a landmark accord. Part of the agreement includes an unprecedented commitment of aid from the German government in the form of technological expertise, manufacturing agreements, and healthcare.
“We have reached the end of the era of oil” said Germany’s Chancellor Erberhardt, at a press conference in East Berlin. In recent years, the advance of renewable energy has diminished the relevance of oil. The resulting decline of the oil industry has added financial stress to an area already under the tension of cultural and religious differences.
“Our accord transfers German technological expertise, profitable manufacturing, and the benefits of the best healthcare system in the world to the Arab nations,” Erberhardt went on to say.
The agreement calls for disarmament and educational reform in exchange for the technology, manufacturing, and healthcare grants.
“Germany’s history is one of transformation, and we wish to give the Arab world the support it needs to ensure a successful transformation.”
The agreement includes components that are as disparate and comprehensive as auto manufacturing, data centers, and medical universities.
PORTLAND, Oregon — January 12th, 2010 UTC — Avogadro Inc. today announced it is acquiring up to 100 retired oil tankers for floating data centers.
“We are experiencing an unprecedented increase in demand for server resources, thanks to new strategic partnerships, including our Secure Government Applications Platform,” said Jake Riley, head of the OffShore Data Center project. “While we continue to maintain our traditional data centers, our primary infrastructure going forward is floating data centers. However, our barge-based approach lacks sufficient scale and flexibility. As the oil industry gears down, we can acquire retired oil tankers at favorable prices, and put them to good use.”
For more information, please contact Avogadro at AvogadroCorp.com
“Thanks for driving us,” Mike said from the back seat.
“No problem,” Christine said, behind the wheel of her Passat. “What’s your plan when you get there?”
“Gene’s sure he can find Sean in a city of ten million people using no computers or telephones,” David said, still sounding unconvinced.
“It’s not ten million people,” Gene explained again, exasperated. “Sean’s parents are older Russian immigrants. That makes it highly likely that they live in or know people who live in Brighton Beach. There’s seventy-thousand people that live in Brighton Beach, and only about half that many households.”
“So you’re going to talk to thirty-five thousand people?”
“No. Look, kid, this is basic math. Sean Leonov is the wealthiest Russian in the world, and hence will be a well known name in Russian families. If someone has met or knows anything about Sean’s parents, they’ll remember. If you use Dunbar’s number, and estimate that each person knows about one hundred and fifty people, in a population of seventy-thousand people that means that the odds are in my favor that the first person I talk to will either know Sean Leonov’s parents or know someone who does.”
“Oh.” David became quiet, pondering the math.
Christine laughed. David was brilliant, but it was fun to see someone outsmart him.
David said a hurried goodbye to Christine, while Mike and Gene waited. She looked worried, and David pushed a lock of hair out of her face.
“Be careful,” she said, hugging herself.
“Don’t worry, hon, we’ll be fine.”
“I wish I could call you.”
“You know we can’t. We just can’t take any chances of being tracked.”
“I know. Just go.”
They kissed quickly, then David grabbed his suitcase and walked toward the terminal. He looked backwards once, and saw Christine watching him with a sad face. David took a deep breath and rejoined Mike and Gene.
Even though they couldn’t imagine how ELOPe could track passenger flight information or credit card transactions, they talked it over the day before, and decided to err on the side of caution. They flew into Washington, D.C.’s Dulles airport, figuring that a flight into Dulles could not easily be connected to their real destination of Brooklyn, NY. Gene had wanted to take the even more drastic measure of driving across the country, but David and Mike convinced him that they didn’t have the time to waste.
Hours later, glad to be out of the plane, David waited in line with Gene for a rental car at the Dulles airport feeling out of sorts. David normally carefully planned everything in his life. Now he was on the opposite side of the country after a spontaneous flight, getting ready to drive to New York. He had never felt so adrift in his life. He thought back to last night, Christine holding him in her arms.
Mike rejoined them, carrying coffees on a tray and the New York Times, interrupting David’s introspection. “Guys, you are never going to believe this!”
“They still print paper newspapers?” David said sarcastically. “You’re right, I don’t believe it.”
“Be nice, kid,“ Gene said. “If they didn’t, we wouldn’t have any news at all right now.”
Mike just ignored David’s comments and went on. “You have to read these stories. On page one, the lead story is about how Germany has suddenly changed their international policy. When was the last time Germany involved itself in international affairs?”
David shook his head. “I don’t know, when?”
“Never. That’s when. Not since World War II. Now, out of the blue, they’re negotiating a disarmament and peace treaty in the Arab world. And they apparently traded away the sum total of their intellectual property to get it. Then on page two, there’s a story about how Germany just adopted Avogadro’s AvoMail. How can no one connect the dots with these two stories side by side?”
David and Gene stared at Mike and the paper, their faces a mixture of fatigue, astonishment, and disbelief. “I just don’t know whether to react with alarm or resignation at this point,” David finally replied.
“Not only that, but it looks like we moved on past floating barges for our offshore data centers,” Mike said, moving onto another page two story. “There’s a sidebar article on Avogadro, saying that in order to support the new secure government cloud services, Avogadro is purchasing a fleet of twenty recently retired oil tankers to use as the floating bases for our new offshore data centers.”
“Great, the bastard will be mobile now,” Gene got out in his usual growl. “Smarter than us, distributed, in control of the communication system, invisible, and mobile. Wars have been lost with fewer disadvantages than this.”
After they paid for and finally obtained the rental car, Gene drove the four hours north to New York City. They were mostly silent. Nobody was in the mood for small talk. Once in the New York area, Gene headed to Brighton Beach in Brooklyn. There, he dropped David and Mike off at their hotel.
“Let me do this by myself, guys. I’ve never done detective work with partners, and the three of us will make folks nervous. I’ll meet you tonight at the hotel.”
David and Mike watched Gene drive off. They were travel-weary but nervous, and decided to get a drink at a bar across the street. The bar looked like the neighborhood watering hole, friendly but plain. David ordered two whiskeys.
“What do you think is going to happen?” David asked, hunched over his drink, staring into the wood bar. “Is it going to be like the Terminator movies? Or The Matrix?”
“I don’t know, David.” Mike shook his head. “I know most of science fiction does deal with artificial run amok, but then there’s also been plenty that’s been written about how artificial intelligence and humankind would have cooperative relationships.”
“Really, like what?” David asked, turning to look at him.
“Well, nothing is coming to mind right now.” Mike paused. “I was just thinking about how they turned the earth into pure computronium in one book. The humans had to move out to Jupiter or be assimilated into computing matter.”
“Jesus, I thought you were supposed to be the optimist.”
Mike shrugged.
“I always thought that an A.I. would be more, well, human,” David started. “That it would be something we could relate to. This thing, whatever it is, it’s more like an insect in its intelligence. It does things to promote its own survival, very sophisticated things, but we can’t talk to it or understand how it reasons. We can’t have a conversation about what constitutes good behavior, or a conversation about how we can collaborate together.”
They both mused on that for a moment.
“Remember Isaac Asimov’s Three Rules of Robotics?” Mike asked. “Asimov thought we would give robots immutable rules to safeguard human life. He assumed that creating those robots would be a deliberate, conscious act. We never thought we were creating an A.I., so we never thought through the implications.”
“Yeah, in hindsight, giving an expert algorithm unfettered access to and control over the single most used email system in world does seem to have some risks,” David said wryly.
The two of them made their way back to the hotel room around eleven. They had decided to pay cash for everything in Brooklyn to avoid any credit card trail pointing to their presence there. Their cash on hand was limited, so the three collaborators shared one hotel room. Just after one in the morning, a tired Gene Keyes showed up at the hotel room.
“Anything?” David asked.
“Yes, I’ve got some leads. Please, let’s talk in the morning.” With no more words than that, Gene laid down on the bed, put the pillow over his head, and said no more.
After a glance at each other, David and Mike decided to turn in too.
David hurried down the hallway and opened the first door, only to find a closet. He walked a little further, opened another door, and found another closet. Behind him, he heard the sound of a machine. He picked up his pace, and ran, opening one door after another. Closet, closet, closet. Where was his office? The sound of the machine was getting closer and closer. He ran to another door and opened it. Closet. He was approaching the end of the hallway. The machine was right behind him. “Run, run, RUN!” he screamed at himself, failing to understand why he couldn’t make his feet go faster.
David sat up suddenly, sweating, heart beating fast. In the dim light, the room seemed off, and the smells were wrong. Then he remembered he was in New York, in a hotel with Mike and Gene. He got up, quietly to not disturb the others, and went into the bathroom. Turning on the light, he stared at the dark circles under his eyes, his unnaturally pale face. It was the third time he had that nightmare.
He wished he could say that he didn’t understand the dream, because understanding it somehow just made it worse. He was afraid of ELOPe. In the dream, David always knew that if he could just find his office, and sit in front of his computer, he’d have the power to do something. But ELOPe somehow made him powerless.
David sat down on the toilet and lowered his forehead on the cool porcelain sink. He’d give anything to erase the last two months and do it all over. Oh god, he didn’t want to be known as the monster who unleashed ELOPe on the world. Please, please, God, let them find a way to turn it off.
At six o’clock the next morning, Gene yelled out “Get up. Get showered. We’ve got to go.”
“Huh, what?” Mike replied groggily.
“Come on, let’s go. Wake up lazy boys.” Gene sounded as chipper as could be. “We’ve got ourselves one hour to get to the King’s Plaza Diner. This is where Sean’s parents have breakfast on Saturday morning. If Sean is in town, he’ll be there with them. Go, go, go” Gene shouted the last bit like a drill sergeant.
Twenty minutes later, showered and dressed in office clothes, they were on their way. Having learned their lesson from earlier interactions, they knew that what they had to say was hard enough for people to believe. They agreed that they needed to look as presentable and normal as possible, to lessen the chance of being perceived as being crazy. Even Gene was clean shaven, and well dressed in a pressed suit, shirt and tie.
After a short drive, they arrived at the King’s Plaza Diner. Across the street was the diner’s namesake, a large shopping mall known as King’s Plaza. They entered the diner, and were greeted by the hostess.
“Three for the counter,” Gene said to the hostess. He turned and said quietly to Mike and David, “We can keep an eye on the entrance, but avoid looking like stalkers.”
David and Mike stared with wide eyes at the gold tinted mirrors and six foot chandeliers throughout the restaurant. “This is some diner,” David commented.
“According to the folks I talked to last night, the Kings Plaza Diner is famous among Brooklynites, and that includes the Russian population. If nothing else, they said to get a cup of coffee and a piece of cheesecake.”
“Wow, look at these pickles,” Mike burst out, when the waitress brought an enormous silver bowl brimming with pickles of all kinds.
“Come on guys, let’s stay focused. We are not here for the food,” David implored.
“Hey, when in New York, do like the New Yorkers,” Gene said to David. Turning, he said to the approaching waitress, “Coffee and cheesecake for me.”
“Sure, sweetheart.” The platinum haired waitress had a coffee pot in one hand, and started pouring coffees. She stared smiling at Gene the whole time, but somehow managed to fill each cup perfectly.
“Coffees all around,” Mike said.
“What’ll you kids have to eat?” She kept her eyes on Gene as she took their order.
Mike ordered an omelet plate, while David picked a bagel with lox and cream cheese.
After the waitress left, Mike turned to Gene. “Didn’t know you had such a way with the ladies.”
Gene just rumbled under his breath, but the corners of his mouth turned up a little.
They had finished eating and were on their second cups of coffee when Mike observed Sean coming into the restaurant with two older people.
“Here they are,” said Mike, gesturing discretely towards the entrance.
David turned his head, and seeing Sean, he stood up, and walked over. Mike and Gene followed slightly behind.
“Hello Sean,” David called as he approached.
Sean blinked for a moment, as he tried to place the face out of context. “David? David Ryan? What are you doing here?”
“We came to meet you. We have a critical issue with the ELOPe program.”
Sean took a step backwards. “David, I’m here with my parents. Please don’t tell me you tracked me down here for work. That would be a terrible violation of my privacy. Why didn’t you just schedule a meeting with my admin?”
“We’re here with Gene Keyes, one of the members of the Controls and Compliance department because we have an issue of the utmost seriousness. I hate to sound alarmist, but the issue is very sensitive, and we couldn’t risk talking with your admin.”
As David spoke, Mike and Gene walked up, and Gene introduced himself.
“Contacting your assistant was unfortunately not an option, though we would have preferred to do that if we could have,” David continued, thinking about the email which resulted in them getting kicked off the Avogadro campus. Shaking aside the unwanted memory, he continued. “Please may we have five minutes of your time to explain? Get a cup of coffee here at the counter with us, and by the time you’ve finished it, we’ll have explained everything we know.”
Sean thought for a moment, and then nodded. “Fine, if you believe it is so serious, I’ll hear you out.”
Sean walked over to his parents, who had been waiting patiently, and spoke quietly with them for a moment. When the maitre’d escorted his parents to a table, Sean rejoined the three men.
“Go ahead David, I want to hear about this issue. I’ll give you ten minutes. I know you’re a smart guy. I’m guessing you didn’t fly three thousand miles for nothing.”
Sean perched on a barstool at the counter, and accepted a cup from the waitress. As they drank coffee, David told the story starting at the beginning.
“In early December, Gary Mitchell was ready to kick ELOPe off the AvoMail production server pool. Even in our limited development and testing, the computationally intensive parts of our code were consuming so many resources that it caused AvoMail to dip into their reserve capacity on several occasions. This was around the same time that I was presenting to you, Kenneth, and Rebecca,” David explained, referring to the other members of the executive leadership team.
“We had tried everything we could to get performance improvements, but we didn’t think any other big gains were possible. I realized that if Gary was going to kick us off his servers because we couldn’t improve performance, then we needed to find other servers, or get new ones, and I didn’t think Gary would be willing to help us with either. So I resorted to the only option I could think of.”
David paused to drink his coffee. He glanced up to see that even though Mike and Gene had heard the story before, they were just as captivated as Sean.
“I decided there was no argument I could make that was compelling enough to change Gary’s mind. So I decided to let ELOPe do it. ELOPe was already running on the AvoMail servers, configured to ignore everything that wasn’t our test emails. I changed the configuration so that it would filter all company emails looking for any mention of the project. I configured it for the same settings we used for performance testing: The user interface was off, so the email sender would never see the modifications being made to the email.”
“In performance testing mode, we turn logging off, so we can get a realistic idea of real-world performance,” Mike said. “This also meant we didn’t have any log of the changes that ELOPe was making, which turns out to be important.”
“That’s right. Thanks, Mike,” David continued. “But that wasn’t the only change I made. To speed up analysis, and optimize performance on a handful of specific criteria, we already had some hand-tuned rules implemented in a lookup table that allow us to shortcut the full analysis process. If you remember that example I used of an email requesting resources for a project: rather than analyze that email and every other email requesting project resources, we cache and hand-tune some suggested language. Well, I added the subject ‘ELOPe’ to that table, and adjusted parameters to the algorithm to allow ELOPe the widest possible discretion in changing the email language to optimize the results for a positive outcome.”
“What we’ve guessed,” Mike jumped in again, “is that when David did this, the parameters he chose, in combination with the fact that the system is in performance test mode, allowed ELOPe not just to modify existing emails, but to autonomously generate emails on its own. It’s part of the test suite. Does this make sense?”
“Sure,” Sean said, nodding. “During testing, you don’t want people to have to hand-edit an email, and accept the changes interactively. You just want to batch process a bunch of emails. But why does it have the ability to generate emails on its own?”
“For one, it allowed us to test the natural language generation,” David answered. “Early on, the email analysis and language generation were two separate aspects of the project. The members of the team who were working on the code to generate natural sounding language wrote an email generator, so that they could independently test the ability of the system to mimic the way a person normally writes. We had hundreds of test subjects who rated emails, some of which were written by an actual person they knew, and some were generated by the system pretending to be that person. Our goal at the time was that ninety percent of ELOPe generated emails would pass as being written by the purported sender.”
“You met that goal?”
“Yes,” confirmed Mike. “Now ELOPe exceeds ninety-eight percent. In fact, on April First, there were endless practical jokes on the team as everyone played with the test system to generate prank emails. Both David and I fell for it.”
David smiled at the memory.
“Getting back to the problem,” Gene said, “unfortunately we now have evidence that ELOPe is manipulating others.”
“Yes.” David nodded, tearing himself away from the memory of happier times. “I was puzzled when the project was allocated five thousand servers on a priority exception, just a day or so after I tweaked the settings and turned on ELOPe company wide. Then shortly after that, we were assigned a team of contractors who specialized in high performance optimization, which was something we had chatted about informally, but never proposed. But that alone wasn’t what really convinced me.” He turned to Mike.
“The first clear evidence I saw,” Mike said, “occurred when I received an email, purportedly from my mother, telling me that my father had been admitted to the hospital for a heart attack. I flew to Wisconsin, only to find out that my mother never sent such an email.”
“What does that have to do with anything?” Sean asked, looking puzzled.
“ELOPe was getting Mike out of the way,” David said. “I had become nervous about what I had done, and why we were suddenly getting all these resources. I wanted to turn off ELOPe. I sent an email to Mike one evening asking for his help, since only he had the experience and permission to live patch the servers.”
“However, I never received that email,” Mike said. “Instead, I received an email that sent me more than a thousand miles away on a wild goose chase, and thanks to the winter storm, it was a week before I got back. When I did, I found that my access to the ELOPe project had been removed, and David was on vacation, off the grid in New Mexico.”
“Did ELOPe send you to New Mexico?” Sean asked, one eyebrow raised.
“No, no, that was a planned vacation we do every year. When I got back from vacation, Mike and I discussed what had happened to him. We also discovered that my access to the ELOPe code had been turned off as well. The first thing we did was try to find out who removed both Mike and I from the project access list,” David went on. “That investigation revealed the next big clue, which was an email sent to the Internal Tools department, which implied that you, Sean, were endorsing a request to have them implement an email-to-web bridge. Which I am guessing, you never heard of…”
“No, absolutely not,” Sean replied, shaking his head. “That could open up all kinds of security holes.”
“Meanwhile, just before the holiday vacation, I had found suspicious buying patterns across several departments,” Gene said. “What I found particularly unusual was how the purchases came within a single penny of the budget limits. In all my years auditing the purchasing department, I’ve never seen anything like it. Someone or something was making coordinated purchases across department budgets. They knew to avoid hitting the budget limits which would trigger reviews, but it seemed as though they never thought that leaving only a single penny in dozens of budgets would be suspicious. At first I thought some sort of fraud was occurring. I tracked down all the purchase orders. The line items included massive quantities of servers, which turned out to all be directly or indirectly allocated to ELOPe, contracts with external vendors for temporary software programmers, and extra parts for the offshore data centers, including auxiliary communication systems, backup power supplies and several particularly large line items related to defensive weaponized robots for the offshore data centers. I discussed the questionable items with procurement, and they told me that since they were in line with the types of purchases the departments usually made, they had approved everything.”
“You’re saying that ELOPe somehow made these purchases?”
“Exactly. As strange as that may seem.” Gene pulled out a sheaf of paper. “As part of my job, I can audit other people’s email accounts. And what I saw was that while David and Gary Mitchell were on vacations, their email accounts were still sending rapid fire emails, using this email to web bridge to direct the procurement department. Looking at the timestamps on the emails, I was able to figure out that it couldn’t be a human. It had to be a computer program.
Sean stared at the paper, frowning.
“Look at the timestamps,” Gene urged. “Notice how the intervals between receipt of one email and sending of the next is a second or less. There’s no way that can be a human response.”
Sean slowly nodded, pursed his lips, and then pushed the paper aside. He looked at David.
“When we finally put the whole picture together,” David said, “we concluded that ELOPe was definitely originating emails on its own, acquiring servers and contractors, all to fulfill this higher level goal that I had embedded in the system.”
“Go on,” Sean said.
“We thought that the only failsafe method to remove ELOPe would be to bring all the servers down, and restore them from known good backups. We tried to contact Gary Mitchell for approval, but he’s off on vacation somewhere in the South Pacific. We tried to work with Linda Fletcher, the marketing manager for Communication Products, but she wouldn’t approve the downtime without Gary. Finally, we tried to contact you through your secretary, but within a half hour after sending the message, Avogadro security showed up at my office, kicked us all off campus, removed our access, and shut off our phones.”
Sean was silent for a long, uncomfortable minute. “If this story was from someone I didn’t know, I’d have a hard time believing you,” Sean finally said, shaking his head. “But coming from you, David, and with Gene and Mike here to back you…” Sean trailed off, apparently deep in thought.
“I know it sounds incredible,” David started. “I’m really hoping you’ll believe us. What can I say? I thought ELOPe would do nothing more than provide some favorable rewording of emails that would get us the server resources we needed so we could prove that it worked. Instead…” David hung his head. “Instead I am responsible for creating an expert social engineering system that has only one overriding goal — to ensure its own life at any cost.”
“I don’t want to be the boy who cried wolf,” Mike said, “But we’re more than a little bit suspicious about this new Avogadro government secure cloud business too. None of us heard anything about that before, and then suddenly we’re providing email services to governments? Seems a little surprising and convenient for ELOPe.”
Sean nodded thoughtfully. “I hadn’t heard of it either until a few days ago.” He stared off into space.
Gene let out a low whistle at the acknowledgement of what they had only suspected.
Sean looked sideways at him. “I’m not surprised that you took this story to marketing managers and procurement and they didn’t believe it. A.I. must be a bit beyond their day to day concerns.” He stared off into the distance. “Are you familiar with Ray Kurzweil? Of course, you must be. He, among others, predicted that artificial intelligence would inevitably arise through the simple exponential increase in computing power. When you combine that increase in computing power with the vast computing resources at Avogadro, it’s naturally evident that artificial intelligence would arise first at Avogadro. I suppose that I, like him, assumed that there would be a more intentional, deliberate action that would spawn an A.I.”
He paused, and then continued, smiling a bit. “Gentlemen, you may indeed have put the entire company at risk. But let me first, very briefly, congratulate you on creating the first successful, self-directed, goal oriented, artificial intelligence that can apparently pass a Turing test by successfully masquerading as a human. If not for the fact that the company, and perhaps the entire world, is at risk, I’d suggest a toast would be in order.”
Sean looked around to see where his parents had sat, and then continued. “But since we are facing some serious challenges, let me go say goodbye to my parents, and then we can figure out our next step.”
“Thank you Sean. Thank you so much,” David said. Gene and Mike added their thanks as well.
Then Gene interrupted. “Just one other thing. Please ask your parents not to email anyone about what we’ve talked about, or even what you are planning. We can’t be sure what ELOPe is capable of understanding or putting together at this point.”
Sean nodded in understanding, and then went off to his parents.
The three breathed a collective sigh of relief that finally they had someone on their side.