Tuesday, November 8, 2022

"Move fast and break things" meets industrial facilities

(updated 10/1/23)


"Start by being nice to every person you meet. But if someone tries to exercise power over you, exercise power over him."
― Nassim Nicholas Taleb, Skin in the Game


This is my account of me quitting my job at CrossnoKaye.

Because of this blog, in November of 2022, the company Board of Directors ordered the management to hire a law firm to investigate the company on ethics and safety concerns. Everything reported here can be verified from the company records (Slack, gmail and google meet recordings). Several employees who have criticized the company direction and approach to safety have left or been let go since. 

In February of 2023, I notified Lineage Logistics, an international food chain supply company that used CrossnoKaye software at their pilot sites (and who was CrossnoKaye's "main source of legitimacy" in the words of an early founder who was forced to leave), of this blog and the investigation. 

On March 9, 2023, three days after my LinkedIn profile was viewed by a high-level Engineering manager from Lineage Logistics, around 2PM PST, in the span of one hour, this blog saw a spike of approximately 100 views coming from Windows platforms located in the US, with multiple references to the blog from Microsoft Teams. I do not know whether the views came from Lineage Logistics discussing the blog in an internal meeting, whether they brought the matter up with CrossnoKaye, or what the outcome was.

On March 29, 2023, CrossnoKaye filed a 120-page petition for restraining order against me, listing this entire blog multiple times as the primary evidence and citing fears for their safety and "safety of their investors and clients" as a reason to request the Court to ban me completely from communicating with any of their investors and clients. At the preliminary hearing on April 24, 2023, I requested the case to be presided by a Judge, and the hearing was held on May 10, 2023, where the Company won the "no contact with the executives" clause of the order, which served to support the gag order; the gag order -- the "no contact with investors and clients" request -- was denied.

On September 8, 2023, an interviewee wrote in a Glassdoor review,  “However, found out today that they did layoffs this week, despite telling me they had "3 years of funding." And "there is job security for the next couple of years." Their excuse for the layoffs was "we overestimated sales." The final interview was more of an interrogation than an interview. Beware! This company sounds like a mess! Took 5 hours out of my work week to interview for a position they knew wasn't going to be filled.”

(The sales model and pricing has never been made clear to the employees but it I inferred it to be at about $50K/site/year. A former employee said that as of early 2023, the 50+ employee company had its product running at about a dozen sites, which would about cover the two founders' salaries. The company appears to have changed it sales tactics since.)

Highlighted below is the initial content of the blog published on August 21, 2022, which the investors and Lineage Logistics were notified about. 


* * *

"When you strike a king, you must kill him" -- Ralph Waldo Emerson

I worked at CrossnoKaye, a Santa Barbara, CA startup, as a Senior Software Engineer, from May 2020 to July 2022. The company describes itself as a "B2B SaaS company building a cloud-connected operating system that will revolutionize the heavy industrial sector". Software as a service in the public cloud with constant updates behind the scenes is great for Netflix, but in an industrial facility a single faulty update or a disconnect from the cloud at the wrong time can cause an industrial accident. But SaaS is how startups like CrossnoKaye wooed VCs back in 2020, which sets the stage for the drama. 

From an employee perspective, this is my summary of CrossnoKaye:

The good: 

  • Opportunity to learn about heavy industries
  • Friendly people (except for the management when challenged)
  • Modern tools and technologies

The bad:

  • Harvard grads and brogrammers calling the shots where industry experience is critical
  • Avoidance of open discussion on serious issues
  • 40+ employees in a startup with no product-market fit

The ugly:

  • The company is just keeping up appearances.


They tried to terminate my employment and failed.

Had I simply resigned without them striking first, it might have been unfair for me to air their dirty laundry, and had their strike succeeded, I might be dismissed as disgruntled. Because the company botched the firing, they opened themselves up for what follows.

* * *

On the morning of Friday July 15, I joined a meeting listed as a 1:1 with my manager, expecting to talk about a team issue I had been raising for weeks. Instead, the CTO (Jesse Crossno) and the COO (Erik Philipp) were there along with the manager who said we are parting ways effective immediately, added it was nice knowing you and signed off. The COO said he was there only as an HR and that the company is offering 4 weeks severance. I confirmed to the CTO that my personal email they have on file is correct and said I am not interested in their offer. When I started telling him what had been going on, he said I am taking notes. After a couple of minutes of that, the COO said let's talk on Monday.

Our contract states it may be terminated by either side upon delivery of written notice. By Sunday evening I had not received one, so shortly before midnight on Sunday I emailed the CTO this resignation notice:

From: Davor Magdic <*******@yahoo.com> 

Date: Sun, Jul 17, 2022, 11:51 PM

Subject: Davor Magdic's resignation from Crossno Kaye

To: Jesse Crossno <*******@crossnokaye.com>

Dear Jesse,

I am hereby notifying you that I am resigning from Crossno Kaye, effective immediately.

Regardless of what Perry may have said on Friday, you and Erik asked me to meet to talk with you on Monday for unspecified reasons. Since I have not received any written notice from the company and I am still legally employed with the company, I am letting you know that I am choosing to end our business relationship unconditionally, right now.

I have many reasons for wanting to end my employment, the first and foremost being that I have seen enough evidence to believe the company is recklessly endangering the national food chain and I do not want to be a part of it, especially after two years of my repeated, ignored and actively suppressed attempts to bring the company’s attention to the fatal flaws in our ill-conceived Atlas architecture.

This is particularly a concern as CK is now moving into B********* [facility] with its "kill plant" where Atlas software may not only cause food spoilage but threaten life and limb. Two weeks ago, by merely turning on W********** [facility] agents for reading, alerts in all other facilities malfunctioned and we had to turn W********** off.

This completely unnecessary, architecturally wrong interdependence of facilities and overdependence on the cloud has caused many incidents in the past: when we turned on R******** [facilty] for read only, O***** [facility] control stopped working. When we fixed O******, R********* stopped reading. Users are all too frequently unable to log in to Atlas to perform essential duties only because of the misguided cloud-centric architecture. When N***** [facility] power outage happened, the agent wiped out all of its data, including OAS I/O licenses (for which the OAS software was entirely unfairly blamed when the fault was squarely on our end, using experimental and now unsupported k3OS software for critical production facilities), because the agent was looking to the cloud first. The next day, the exact same thing happened in R********.

When I brought up our architectural design issues on Slack back in March, Adam, Perry, and yourself actively suppressed further discussion at the company level, according to Perry, who in a DM first promised to D*** and me a discussion on the issue, which I then announced in the channel, only to have it cancelled next week by Perry saying "I fear if not framed correctly it will seed the perception that we're going in the wrong direction as a company". Adam claimed it was him who told Perry to cancel it, Perry said it was first him and then you who decided to cancel it. This cancellation happened despite my repeated insistence that the worst thing we can do about a problem is not talk about it. All Ops people stood strongly in favor of decoupling the agent from the cloud as I suggested -- turning Atlas from cloud-controlled to cloud-assisted -- and C**** later reused my slides to rekindle the discussion, which to my knowledge still has not gone anywhere.

In addition to suppressing architectural discussions, I have seen what looked like an intentional effort to mislead investors. In the practice pitch for the second round of funding you and Bryan gave before the company, when you touched on user referencability -- a key metric for the Product-Market Fit -- you just said "our users love Atlas" and moved on the next slide despite the fact that our Net Promoter Score was only * [number]  -- our users were actually detractors. Whether that pitch was just for the employees or for the investors as well I do not know, I hope they were told the truth. When P*** [user] in O***** [facility] is recorded on the video not being able to log in to Atlas saying "Here I’ll switch to Atlas platform, or dashboard, or whatever the hell you want to call it", it is clear Atlas is hardly referencable from the user’s view.

The pattern of ignoring problems exists elsewhere in the company. When N****** gave a tech talk on B**** S********* [procedure], indicating that a PMF hypothesis is that customers will want to buy Atlas if it can save them energy and proposing to do this through ****** ******** ****** ***** **** ***** [procedure], I asked if we know if that can affect food quality and no one knew or has reported to have looked into since. When in a DM to N****** I pointed out research that shows high fluctuations in frozen food temperature degrade the quality of the food, with +-5C degrading dough and potatoes significantly (ice crystals form) and just +-2C degrading frozen beef with thaw loss and other undesirable changes, I asked him what temperature fluctuations we expect during B**** S********** and never received a response.

This is to say nothing of the team dynamic problems caused by the bullying behavior of team members and Perry’s inability or unwillingness to resolve it. You are aware of all the damage caused by Drew and how only after my repeated efforts to protect myself and others from his behavior Perry was forced to ask Drew to apologize which led to other people complaining and then Drew being fired -- only to result in Perry effectively reprimanding me for doing what he failed to do. Now a similar situation emerged with Adam, who engaged in disrespectful and manipulative behavior and even badmouthed D*** and C**** to me to try to diminish my pointing out that they too believe we are too focused on the cloud. The attempts at strongarming, disrespect and dishonesty became so unbearable with Perry allowing or condoning it that I felt I had no choice but to ask that he resolve this, which he never did. Of course I continued working with Ops and other people, even through a national holiday, to enable W********** and kept monitoring the facilities and providing support. I also kept meeting with M**** to help him with onboarding -- I was against him being hired for his own protection as I would not have recommended to anyone to work with Roman and Adam whose disrespect I documented in detail in DM to Perry.

The very fact that you didn’t know what was going on, on your own admission, shows that my concerns about Perry retaliating were justified.

I will only add this: four months into my employment, J**** C********* reached out to me to meet for coffee. He talked about enjoying not working at all -- there was never a project he said he was leaving CK for. He told me he left the company because his sense was that it was playing fast and loose with the safety and reliability of critical systems.

In retrospect, I wish I had listened to him and quit earlier myself, but it is never too late.

Sincerely,

Davor

Immediately upon sending the email I texted the CTO to let him know about my resignation. Their response Monday morning and in the days that followed seemed one of confusion and panic. First they insisted they terminated me (and threw in a "we take safety very seriously" boilerplate response in their reply), then wavered when I quoted the contract termination clause, then went back the next day to insisting it was termination and not resignation, sending me multiple notices on their own and asking me to disregard previous ones, I assume to escape filing my notice as a legal document that investors might see. I did not ask anything of the company, and by the terms of the contract their acceptance of my resignation was not required.

For the record, two months earlier, on my second annual review, I received great reviews from coworkers and 11% raise. (On my first annual review, where Perry, my manager, gave me the negative feedback for defending myself, I received equally good reviews and 3% raise.)

If everyone is respectful and I disagree with the company direction, I wish them good luck and leave. But if someone in a position of power tries to disrespect me repeatedly and the management allows it, I don’t leave; I have done nothing wrong. Since they decided to just get rid of me and couldn’t even do that -- who goes to fire someone unprepared? -- I believe it fair to make my experience public.

                                                               * * *

A version of this letter has also been posted on the company's Glassdoor

It took the company weeks to respond after the review was posted, yet their response consists entirely of platitudes: "It’s very important to our company that we keep an open and public dialogue on the path our technology takes and make thoughtful and safe decisions". It does not address a single point raised.

(They almost certainly knew about the blog and the GD review from the day it was published -- initially I put the blog url in my LinkedIn profile under my CrossnoKaye experience and, as chance would have it, the next day I saw that another C-level person from the company viewed my profile. That same day I saw in the blog stats that the blog was referred to from Slack and from Gmail. There being practically no other views to the blog, it must have been the CrossnoKaye management who saw it and shared between themselves.) 

There is only one claim from my resignation letter that cannot be verified from electronic records, and that is my meeting with J**** C*********, who later told me he voiced much the same concerns to Erik when he left.

                                                               * * *

To clarify, now that the blog seems to be getting a wider audience after CrossnoKaye filed for a restraining order (really an attempt at a gag order) --

The company had all the rights to move to terminate me, even if they failed to execute on it, because my employment was "at-will". There was nothing illegal there; I would be the first to defend their right to fire me at any time, if for no other reason than so that I have my right to resign at any time. This was not a wrongful termination because it was not a termination. I resigned, and I was free -- free from the bad job, free from attempts at disrespect I had to fight off constantly, free from the bad management, and free to tell the world what happened (while honoring my employment agreement to keep any proprietary information confidential). It was exhilarating, and it would not have been possible if they had not tried to terminate me. (I do miss some of the people I worked with but who knows what the future holds, we might work together somewhere else.)

The company's attempts to claim they terminated me are just silly: if you have to fight that battle, you've already lost. Most companies would welcome with open arms that an employee who they want out resigns, for countless reasons. To fight tooth and nail to try to convince the world you fired the employee, and against the contract so clearly written to boot, means you have something to hide.

(One exception to "full rights" is if they tried to fire me because of my repeatedly bringing up safety issues, which then might qualify as retaliation for whistleblowing, but I am not a legal expert. And even so: I wasn't fired, I quit my job at CrossnoKaye.)

Forgive this personal touch, I will now go back to the facts of the matter.

                                                               * * *

My LinkedIn profile is at linkedin.com/in/davor-magdic.

                                                               * * *

The following few items describe different aspects of CrossnoKaye management's inability and unwillingness to handle serious problems; the blog then moves on to chronological updates.

It may be a lot to take in, but given the seriousness of what the company is involved in, I am erring on providing more context.

                                                               * * *

"Davor should have handled Drew through management."

This is how Perry, my boss, effectively reprimanded me in the form of "anonymous" negative feedback which he used to close my first annual review held two weeks after Drew, the bully, was fired following my public confrontation with him after months of management inaction. What follows is the account of how we got there.

Drew got the job as an inside connection, without going through a coding test everyone else had to pass. From the day he started he had been praised by the management as brilliant and was immediately taken into leadership role. He became abrasive fairly quickly which seemed to only further assure the management they were dealing with a genius.

A month into the job he interrupts my report at a standup, unmentioned, unexpected, unprovoked, snapping at me "why the fuck would you do that?!" I respond back with a small zing. ("Because in theory, theory and practice are the same, but in practice they are not", I said, but the only correct response would have been "Drew, don't talk to me like that". I was too shocked, as were others; I had to learn on the job.) When the team lead says let's all act respectfully, Drew says “You mean like not saying fuck?”, raises two thumbs up and grins, “Fucking A!”

Eventually I realized that was not madness but his tactics: you say anything Drew doesn't like, you get verbally slapped in the face.

In the months that followed I and others experienced variations of that, but the management kept praising him, making him seem untouchable. Everyone became anxious around the guy, even afraid to use words Drew didn't like such as "integration tests" and "drivers", even when he wasn't presentIn multiple conversations I brought up with the management that people lived in fear of saying anything that might provoke Drew, and Erik the COO said to me in a 1:1, "of all the things in this company that worries me the most." Yet nothing changed. (Why didn't he do something about it?) 

Finally, upon another incident I confronted Drew in public. Perry tried to hush everything up but people DMed me with thumbs up in support, saying in their previous jobs Drew would have been fired on the spot. 

At that point Perry had no choice but to ask Drew to apologize; I insisted the apology had to be made before the same people where the offense was made. Perry agreed and this is how it went: at the standup Perry asks Drew "please, Drew, if you'd like, would you apologize to J***". HR and CTO are present. Drew apologizes with "I didn't know I was doing anything wrong, next time, if I am, tell me." Perry asks Drew to apologize to me too, but Drew doesn't, which Perry does not address. Perry then praises Drew how great he is to work with and how much he has contributed to the company. Another person from the leadership joins in on the praise.

(What Drew actually did, in the words of one engineer, no one knew; the first project Drew had worked on for months he decided to scrap, and all throughout he actually blocked all progress on testing a critical piece of software for unknown reasons.) 

That was Friday morning; the next Monday we were told that Drew got the boot, and soon after Perry admitted in a 1:1 it was because after the standup a number of other employees complained about their own bad experiences with Drew.

A week later came my 1st annual review where Perry gave me that feedback that I should have handled Drew through management. To add insult to injury, he hid behind the anonymity (for the receiver) of the feedback process but it is clear the comment came from the management: a non-management person could not have known how much I engaged the management. 

To make matters worse, Perry saved this "feedback" for the very end of the review, just before he would announce my 3% raise.

(In the interest of fairness: neither Perry nor the CTO Crossno nor the COO Erik ever said to me anything untoward; our interaction was always respectful. It is what they did, as well that what they didn't do, that reveals what their character is like under pressure.)

                                                               * * *

The incident recorded in this blog entry describes the above mentioned attempts at strongarming, disrespect and dishonesty in a team lead by Adam and managed by Perry. I'm posting it for the coworkers who experienced the same, whether still at the company or left or fired by now, to know that, as with Drew, you are not imagining what Adam tries to pull off and how Perry plays along.

                                                               * * *

To this day their careers site (and this 11/23 archive.org snapshot) says "Current control systems simply automate what people do by hand without adding intelligence, much like how the first cell phones replaced the functionality of a landline but provided little else." (Emph. mine.) When I first read it I said to the recruiter it sounded condescending and asked her why she wrote it so; she said she didn't, she just picked it up from Jesse and Bryan. I said it sounds like we're putting those people down, we should rewrite it, she (and Perry, to his credit) agreed and she said she would talk to them, but nothing changed.

                                                               * * *

Their website says their software "updates continuously so every site has the latest and greatest features and functionality, always", as if latest features, not reliability and safety, are an industrial facility's primary concern. This attitude is also reflected in how much focus the company puts on making the controls UI look pretty rather than making sure the operators are always able to log in. Engineering users' loyalty is earned with solid functionality, not UI candy.

                                                               * * *

A number of cold storage facilities use an old but reliable software package called Technisystems to control the facility. While CrossnoKaye's ATLAS software has modern data collection and storage on the cloud (and modern UI), at the time I left it did not have parity with Technisystems on key functionality. And the Technisystems software appears to have been written by one person.

ATLAS was written over 2+ years by a dozen or more software engineers. That it could not meet the critical functionality of far more resourcefully built software that they hope to replace shows the necessary engineering experience is lacking and the focus is in the wrong place.

                                                               * * *

When Lineage Logistics, a global food supply chain company, published an article that says CrossnoKaye software is deployed at two of Lineage’s facilities and is expected to be deployed across additional facilities in Lineage’s network, the CTO (Jesse Crossno) commented "whelp, there goes the stealth mode". The "stealth" mode that CrossnoKaye tries to maintain is in my view counterproductive, given that an industrial control system cannot be just rolled out one day, it has to involve a lot of people over a long period of time with many careful iterations. 

                                                               * * *

I want to make it clear that the person who I notified about concerns for food quality degradation with respect to the potential energy saving procedure is a person of integrity and was known for voicing concerns to the management. That person is no longer with the company.

                                                               * * *

A fair question would be, if the VCs, who are not engineers, have been lead into thinking CrossnoKaye SaaS is a good fit for heavy industries, why did I (and other engineers) buy into the story, even ignoring the first warning flag that the founders of a technology company are non-technical? My answer is, there is an element of truth in their claims: the heavy industries do need to modernize their software for greater productivity, and there is an opportunity to reduce the energy consumption. Their story is also nicely packaged and hits all the right notes -- sustainability, saving people money, helping climate. The holes in that story started to appear early on but having had no experience in heavy industries, like literally everyone else who worked on software at CrossnoKaye, I thought I did not see the big picture. Eventually though, the big picture emerged and it was clear that their premise of a Saas-based "operating system" for heavy industries in the cloud was wrong; what I did not expect was that they would not want to discuss it. (The irony is that one of their stated principles, which they kept repeating in all-hands meetings, is "challenge assumptions" -- they even say it on their March 2023 "innovation summit" page.)

Most of all, however, I bought into the story because I was sure the people who gave this startup millions of dollars must have done their due diligence. 

Which, to be fair, they might have, to the degree they could since they are non-technical, seeing how they hired a law firm to investigate engineering problems. The blame may rest simply on the climate that put pressure on the VCs to invest in SaaS companies for 100x returns. As the Theranos whistleblower said"I'm under pressure to exaggerate technology claims, exaggerate revenue projection claims. Sometimes investors will straight-up tell you, you need to double, quadruple, or 10x any revenue projection you think is realistic. I could see how this environment could create an Elizabeth Holmes." 

                                                               * * *

The pattern of half truths was there from the beginning. The recruiter who approached me before I applied in March of 2020 (different from the recruiter I mentioned above) said CrossnoKaye "have been able to bootstrap the entire project without VC funding and have generated over $5M in revenue this year", which sounds like profitability but -- to the degree it is even true -- turned out to be a one-off science project for which founders were hired separately from their company's product. As far as I can tell, the company is nowhere close to being profitable nor does it even have any clear path to profitability. 

Another thing the recruiter said in the initial email was that CrossnoKaye "is reducing energy consumption and costs from the world's biggest polluters by as much as 40%." I don't know if the 40% is repeatable across sites and not just another one-off under artificial conditions, but CrossnoKaye works only with cold storage facilities which are hardly "the world's biggest polluters". This claim, like their other ones, turned out to be all puffery and no substance.

                                                               * * *

All the information presented in this blog is either public, or, like the "millions of dollars" in funding, received outside of my employment with the company (the recruiter said in his email, "they are currently in the middle of a Series A raise targeting $20M at an $80M valuation"), or is my experience and memory interacting with people at CrossnoKaye, or is owned by me as is the case with my resignation letter which I sent from my personal email -- a legitimate means for expressing one party's will in writing for the purpose of ending a business contract -- and from which I redacted any information that could be considered proprietary or pertaining to the company's competitive advantage, per my employment contract. (I also redacted the names of people who were not at fault in any way.) 

Of course, I did not sign anything upon leaving the company. I did however exercise 10,000 shares, at 7c per share, from my employee stock options after I left.

                                                               * * *

Update 10/28/22: I learned that in early October one controls engineer who was a strong voice for safety, and who had been questioning the too-much-SaaS approach, was fired. I consider this engineer a person of integrity, so seeing that the management apparently decided to double down on eliminating dissent, I contacted members of the CrossnoKaye board who are part of the investment group that funded the company (IGSB) to let them know what was happening and pointed them to this blog. The response I received was that they are taking this issue very seriously and are investigating the matter.

                                                               * * *

Update 11/22/22: Erik Philipp, the COO, called me to tell me that my email to the Board has been shared with the company management, causing quite a stir he said, and that the Board had told them to hire an independent law firm to investigate the actions of the company. I was asked for permission to have my contact info shared with this law firm so they can meet with me, which I agreed to. I am not required to keep confidential any of the information I received, but, as a sign of appreciation for the lead investigator's work, I won't post updates until the investigation is completed.

                                                               * * *

Update 2/1/23: The investigation appears to be wrapping up. The law firm that investigated the company is Cooley LLP and the lead investigator who interviewed me on December 2nd is Randall Lee. Updates soon.

Meanwhile I have learned that at least four more employees have left on concerns similar to mine or been fired for voicing them. 

                                                               * * *

Update 3/12/23: Lineage Logistics, the international cold storage and food supply chain company mentioned above which has been using CrossoKaye software at pilot sites and which reported in December of 2022 that it aimed to deploy the software "at over twenty facilities in the next year alone" has been notified of this blog and of the investigation.

No comments:

Post a Comment

"Move fast and break things" meets industrial facilities

(updated 10/1/23) "Start by being nice to every person you meet. But if someone tries to exercise power over you, exercise power over h...