"Move fast and break things" meets industrial facilities

(updated 10/1/23)

"Start by being nice to every person you meet. But if someone tries to exercise power over you, exercise power over him."
― Nassim Nicholas Taleb, Skin in the Game

This is my account of me quitting my job at CrossnoKaye.

Because of this blog, in November of 2022, the company Board of Directors ordered the management to hire a law firm to investigate the company on ethics and safety concerns. Everything reported here can be verified from the company records (Slack, gmail and google meet recordings). Several employees who have criticized the company direction and approach to safety have left or been let go since. 

In February of 2023, I notified Lineage Logistics, an international food chain supply company that used CrossnoKaye software at their pilot sites (and who was CrossnoKaye's "main source of legitimacy" in the words of an early founder who was forced to leave), of this blog and the investigation. 

On March 9, 2023, three days after my LinkedIn profile was viewed by a high-level Engineering manager from Lineage Logistics, around 2PM PST, in the span of one hour, this blog saw a spike of approximately 100 views coming from Windows platforms located in the US, with multiple references to the blog from Microsoft Teams. I do not know whether the views came from Lineage Logistics discussing the blog in an internal meeting, whether they brought the matter up with CrossnoKaye, or what the outcome was.

On March 29, 2023, CrossnoKaye filed a 120-page petition for restraining order against me, listing this entire blog multiple times as the primary evidence and citing fears for their safety and "safety of their investors and clients" as a reason to request the Court to ban me completely from communicating with any of their investors and clients. At the preliminary hearing on April 24, 2023, I requested the case to be presided by a Judge, and the hearing was held on May 10, 2023, where the Company won the "no contact with the executives" clause of the order, which served to support the gag order; the gag order -- the "no contact with investors and clients" request -- was denied.

On September 8, 2023, an interviewee wrote in a Glassdoor review,  “However, found out today that they did layoffs this week, despite telling me they had "3 years of funding." And "there is job security for the next couple of years." Their excuse for the layoffs was "we overestimated sales." The final interview was more of an interrogation than an interview. Beware! This company sounds like a mess! Took 5 hours out of my work week to interview for a position they knew wasn't going to be filled.”

(The sales model and pricing has never been made clear to the employees but it I inferred it to be at about $50K/site/year. A former employee said that as of early 2023, the 50+ employee company had its product running at about a dozen sites, which would about cover the two founders' salaries. The company appears to have changed it sales tactics since.)

Highlighted below is the initial content of the blog published on August 21, 2022, which the investors and Lineage Logistics were notified about. 

* * *

"When you strike a king, you must kill him" -- Ralph Waldo Emerson

I worked at CrossnoKaye, a Santa Barbara, CA startup, as a Senior Software Engineer, from May 2020 to July 2022. The company describes itself as a "B2B SaaS company building a cloud-connected operating system that will revolutionize the heavy industrial sector". Software as a service in the public cloud with constant updates behind the scenes is great for Netflix, but in an industrial facility a single faulty update or a disconnect from the cloud at the wrong time can cause an industrial accident. But SaaS is how startups like CrossnoKaye wooed VCs back in 2020, which sets the stage for the drama. 

From an employee perspective, this is my summary of CrossnoKaye:

The good: 

• Opportunity to learn about heavy industries
• Friendly people (except for the management when challenged)
• Modern tools and technologies

The bad:

• Harvard grads and brogrammers calling the shots where industry experience is critical
• Avoidance of open discussion on serious issues
• 40+ employees in a startup with no product-market fit

The ugly:

• The company is just keeping up appearances.

They tried to terminate my employment and failed.

Had I simply resigned without them striking first, it might have been unfair for me to air their dirty laundry, and had their strike succeeded, I might be dismissed as disgruntled. Because the company botched the firing, they opened themselves up for what follows.

* * *

On the morning of Friday July 15, I joined a meeting listed as a 1:1 with my manager, expecting to talk about a team issue I had been raising for weeks. Instead, the CTO (Jesse Crossno) and the COO (Erik Philipp) were there along with the manager who said we are parting ways effective immediately, added it was nice knowing you and signed off. The COO said he was there only as an HR and that the company is offering 4 weeks severance. I confirmed to the CTO that my personal email they have on file is correct and said I am not interested in their offer. When I started telling him what had been going on, he said I am taking notes. After a couple of minutes of that, the COO said let's talk on Monday.

Our contract states it may be terminated by either side upon delivery of written notice. By Sunday evening I had not received one, so shortly before midnight on Sunday I emailed the CTO this resignation notice:

From: Davor Magdic <*******@yahoo.com> 

Date: Sun, Jul 17, 2022, 11:51 PM

Subject: Davor Magdic's resignation from Crossno Kaye

To: Jesse Crossno <*******@crossnokaye.com>

Dear Jesse,

I am hereby notifying you that I am resigning from Crossno Kaye, effective immediately.

Regardless of what Perry may have said on Friday, you and Erik asked me to meet to talk with you on Monday for unspecified reasons. Since I have not received any written notice from the company and I am still legally employed with the company, I am letting you know that I am choosing to end our business relationship unconditionally, right now.

I have many reasons for wanting to end my employment, the first and foremost being that I have seen enough evidence to believe the company is recklessly endangering the national food chain and I do not want to be a part of it, especially after two years of my repeated, ignored and actively suppressed attempts to bring the company’s attention to the fatal flaws in our ill-conceived Atlas architecture.

This is particularly a concern as CK is now moving into B********* [facility] with its "kill plant" where Atlas software may not only cause food spoilage but threaten life and limb. Two weeks ago, by merely turning on W********** [facility] agents for reading, alerts in all other facilities malfunctioned and we had to turn W********** off.

This completely unnecessary, architecturally wrong interdependence of facilities and overdependence on the cloud has caused many incidents in the past: when we turned on R******** [facilty] for read only, O***** [facility] control stopped working. When we fixed O******, R********* stopped reading. Users are all too frequently unable to log in to Atlas to perform essential duties only because of the misguided cloud-centric architecture. When N***** [facility] power outage happened, the agent wiped out all of its data, including OAS I/O licenses (for which the OAS software was entirely unfairly blamed when the fault was squarely on our end, using experimental and now unsupported k3OS software for critical production facilities), because the agent was looking to the cloud first. The next day, the exact same thing happened in R********.

When I brought up our architectural design issues on Slack back in March, Adam, Perry, and yourself actively suppressed further discussion at the company level, according to Perry, who in a DM first promised to D*** and me a discussion on the issue, which I then announced in the channel, only to have it cancelled next week by Perry saying "I fear if not framed correctly it will seed the perception that we're going in the wrong direction as a company". Adam claimed it was him who told Perry to cancel it, Perry said it was first him and then you who decided to cancel it. This cancellation happened despite my repeated insistence that the worst thing we can do about a problem is not talk about it. All Ops people stood strongly in favor of decoupling the agent from the cloud as I suggested -- turning Atlas from cloud-controlled to cloud-assisted -- and C**** later reused my slides to rekindle the discussion, which to my knowledge still has not gone anywhere.

In addition to suppressing architectural discussions, I have seen what looked like an intentional effort to mislead investors. In the practice pitch for the second round of funding you and Bryan gave before the company, when you touched on user referencability -- a key metric for the Product-Market Fit -- you just said "our users love Atlas" and moved on the next slide despite the fact that our Net Promoter Score was only * [number]  -- our users were actually detractors. Whether that pitch was just for the employees or for the investors as well I do not know, I hope they were told the truth. When P*** [user] in O***** [facility] is recorded on the video not being able to log in to Atlas saying "Here I’ll switch to Atlas platform, or dashboard, or whatever the hell you want to call it", it is clear Atlas is hardly referencable from the user’s view.

The pattern of ignoring problems exists elsewhere in the company. When N****** gave a tech talk on B**** S********* [procedure], indicating that a PMF hypothesis is that customers will want to buy Atlas if it can save them energy and proposing to do this through ****** ******** ****** ***** **** ***** [procedure], I asked if we know if that can affect food quality and no one knew or has reported to have looked into since. When in a DM to N****** I pointed out research that shows high fluctuations in frozen food temperature degrade the quality of the food, with +-5C degrading dough and potatoes significantly (ice crystals form) and just +-2C degrading frozen beef with thaw loss and other undesirable changes, I asked him what temperature fluctuations we expect during B**** S********** and never received a response.

This is to say nothing of the team dynamic problems caused by the bullying behavior of team members and Perry’s inability or unwillingness to resolve it. You are aware of all the damage caused by Drew and how only after my repeated efforts to protect myself and others from his behavior Perry was forced to ask Drew to apologize which led to other people complaining and then Drew being fired -- only to result in Perry effectively reprimanding me for doing what he failed to do. Now a similar situation emerged with Adam, who engaged in disrespectful and manipulative behavior and even badmouthed D*** and C**** to me to try to diminish my pointing out that they too believe we are too focused on the cloud. The attempts at strongarming, disrespect and dishonesty became so unbearable with Perry allowing or condoning it that I felt I had no choice but to ask that he resolve this, which he never did. Of course I continued working with Ops and other people, even through a national holiday, to enable W********** and kept monitoring the facilities and providing support. I also kept meeting with M**** to help him with onboarding -- I was against him being hired for his own protection as I would not have recommended to anyone to work with Roman and Adam whose disrespect I documented in detail in DM to Perry.

The very fact that you didn’t know what was going on, on your own admission, shows that my concerns about Perry retaliating were justified.

I will only add this: four months into my employment, J**** C********* reached out to me to meet for coffee. He talked about enjoying not working at all -- there was never a project he said he was leaving CK for. He told me he left the company because his sense was that it was playing fast and loose with the safety and reliability of critical systems.

In retrospect, I wish I had listened to him and quit earlier myself, but it is never too late.

Sincerely,

Davor

Immediately upon sending the email I texted the CTO to let him know about my resignation. Their response Monday morning and in the days that followed seemed one of confusion and panic. First they insisted they terminated me (and threw in a "we take safety very seriously" boilerplate response in their reply), then wavered when I quoted the contract termination clause, then went back the next day to insisting it was termination and not resignation, sending me multiple notices on their own and asking me to disregard previous ones, I assume to escape filing my notice as a legal document that investors might see. I did not ask anything of the company, and by the terms of the contract their acceptance of my resignation was not required.

For the record, two months earlier, on my second annual review, I received great reviews from coworkers and 11% raise. (On my first annual review, where Perry, my manager, gave me the negative feedback for defending myself, I received equally good reviews and 3% raise.)

If everyone is respectful and I disagree with the company direction, I wish them good luck and leave. But if someone in a position of power tries to disrespect me repeatedly and the management allows it, I don’t leave; I have done nothing wrong. Since they decided to just get rid of me and couldn’t even do that -- who goes to fire someone unprepared? -- I believe it fair to make my experience public.

                                                               * * *

A version of this letter has also been posted on the company's Glassdoor. 

It took the company weeks to respond after the review was posted, yet their response consists entirely of platitudes: "It’s very important to our company that we keep an open and public dialogue on the path our technology takes and make thoughtful and safe decisions". It does not address a single point raised.

(They almost certainly knew about the blog and the GD review from the day it was published -- initially I put the blog url in my LinkedIn profile under my CrossnoKaye experience and, as chance would have it, the next day I saw that another C-level person from the company viewed my profile. That same day I saw in the blog stats that the blog was referred to from Slack and from Gmail. There being practically no other views to the blog, it must have been the CrossnoKaye management who saw it and shared between themselves.) 

There is only one claim from my resignation letter that cannot be verified from electronic records, and that is my meeting with J**** C*********, who later told me he voiced much the same concerns to Erik when he left.

                                                               * * *

To clarify, now that the blog seems to be getting a wider audience after CrossnoKaye filed for a restraining order (really an attempt at a gag order) --

The company had all the rights to move to terminate me, even if they failed to execute on it, because my employment was "at-will". There was nothing illegal there; I would be the first to defend their right to fire me at any time, if for no other reason than so that I have my right to resign at any time. This was not a wrongful termination because it was not a termination. I resigned, and I was free -- free from the bad job, free from attempts at disrespect I had to fight off constantly, free from the bad management, and free to tell the world what happened (while honoring my employment agreement to keep any proprietary information confidential). It was exhilarating, and it would not have been possible if they had not tried to terminate me. (I do miss some of the people I worked with but who knows what the future holds, we might work together somewhere else.)

The company's attempts to claim they terminated me are just silly: if you have to fight that battle, you've already lost. Most companies would welcome with open arms that an employee who they want out resigns, for countless reasons. To fight tooth and nail to try to convince the world you fired the employee, and against the contract so clearly written to boot, means you have something to hide.

(One exception to "full rights" is if they tried to fire me because of my repeatedly bringing up safety issues, which then might qualify as retaliation for whistleblowing, but I am not a legal expert. And even so: I wasn't fired, I quit my job at CrossnoKaye.)

Forgive this personal touch, I will now go back to the facts of the matter.

                                                               * * *

My LinkedIn profile is at linkedin.com/in/davor-magdic.

                                                               * * *

The following few items describe different aspects of CrossnoKaye management's inability and unwillingness to handle serious problems; the blog then moves on to chronological updates.

It may be a lot to take in, but given the seriousness of what the company is involved in, I am erring on providing more context.

                                                               * * *

"Davor should have handled Drew through management."

This is how Perry, my boss, effectively reprimanded me in the form of "anonymous" negative feedback which he used to close my first annual review held two weeks after Drew, the bully, was fired following my public confrontation with him after months of management inaction. What follows is the account of how we got there.

Drew got the job as an inside connection, without going through a coding test everyone else had to pass. From the day he started he had been praised by the management as brilliant and was immediately taken into leadership role. He became abrasive fairly quickly which seemed to only further assure the management they were dealing with a genius.

A month into the job he interrupts my report at a standup, unmentioned, unexpected, unprovoked, snapping at me "why the fuck would you do that?!" I respond back with a small zing. ("Because in theory, theory and practice are the same, but in practice they are not", I said, but the only correct response would have been "Drew, don't talk to me like that". I was too shocked, as were others; I had to learn on the job.) When the team lead says let's all act respectfully, Drew says “You mean like not saying fuck?”, raises two thumbs up and grins, “Fucking A!”

Eventually I realized that was not madness but his tactics: you say anything Drew doesn't like, you get verbally slapped in the face.

In the months that followed I and others experienced variations of that, but the management kept praising him, making him seem untouchable. Everyone became anxious around the guy, even afraid to use words Drew didn't like such as "integration tests" and "drivers", even when he wasn't present. In multiple conversations I brought up with the management that people lived in fear of saying anything that might provoke Drew, and Erik the COO said to me in a 1:1, "of all the things in this company that worries me the most." Yet nothing changed. (Why didn't he do something about it?) 

Finally, upon another incident I confronted Drew in public. Perry tried to hush everything up but people DMed me with thumbs up in support, saying in their previous jobs Drew would have been fired on the spot. 

At that point Perry had no choice but to ask Drew to apologize; I insisted the apology had to be made before the same people where the offense was made. Perry agreed and this is how it went: at the standup Perry asks Drew "please, Drew, if you'd like, would you apologize to J***". HR and CTO are present. Drew apologizes with "I didn't know I was doing anything wrong, next time, if I am, tell me." Perry asks Drew to apologize to me too, but Drew doesn't, which Perry does not address. Perry then praises Drew how great he is to work with and how much he has contributed to the company. Another person from the leadership joins in on the praise.

(What Drew actually did, in the words of one engineer, no one knew; the first project Drew had worked on for months he decided to scrap, and all throughout he actually blocked all progress on testing a critical piece of software for unknown reasons.) 

That was Friday morning; the next Monday we were told that Drew got the boot, and soon after Perry admitted in a 1:1 it was because after the standup a number of other employees complained about their own bad experiences with Drew.

A week later came my 1st annual review where Perry gave me that feedback that I should have handled Drew through management. To add insult to injury, he hid behind the anonymity (for the receiver) of the feedback process but it is clear the comment came from the management: a non-management person could not have known how much I engaged the management. 

To make matters worse, Perry saved this "feedback" for the very end of the review, just before he would announce my 3% raise.

(In the interest of fairness: neither Perry nor the CTO Crossno nor the COO Erik ever said to me anything untoward; our interaction was always respectful. It is what they did, as well that what they didn't do, that reveals what their character is like under pressure.)

                                                               * * *

The incident recorded in this blog entry describes the above mentioned attempts at strongarming, disrespect and dishonesty in a team lead by Adam and managed by Perry. I'm posting it for the coworkers who experienced the same, whether still at the company or left or fired by now, to know that, as with Drew, you are not imagining what Adam tries to pull off and how Perry plays along.

                                                               * * *

To this day their careers site (and this 11/23 archive.org snapshot) says "Current control systems simply automate what people do by hand without adding intelligence, much like how the first cell phones replaced the functionality of a landline but provided little else." (Emph. mine.) When I first read it I said to the recruiter it sounded condescending and asked her why she wrote it so; she said she didn't, she just picked it up from Jesse and Bryan. I said it sounds like we're putting those people down, we should rewrite it, she (and Perry, to his credit) agreed and she said she would talk to them, but nothing changed.

                                                               * * *

Their website says their software "updates continuously so every site has the latest and greatest features and functionality, always", as if latest features, not reliability and safety, are an industrial facility's primary concern. This attitude is also reflected in how much focus the company puts on making the controls UI look pretty rather than making sure the operators are always able to log in. Engineering users' loyalty is earned with solid functionality, not UI candy.

                                                               * * *

A number of cold storage facilities use an old but reliable software package called Technisystems to control the facility. While CrossnoKaye's ATLAS software has modern data collection and storage on the cloud (and modern UI), at the time I left it did not have parity with Technisystems on key functionality. And the Technisystems software appears to have been written by one person.

ATLAS was written over 2+ years by a dozen or more software engineers. That it could not meet the critical functionality of far more resourcefully built software that they hope to replace shows the necessary engineering experience is lacking and the focus is in the wrong place.

                                                               * * *

When Lineage Logistics, a global food supply chain company, published an article that says CrossnoKaye software is deployed at two of Lineage’s facilities and is expected to be deployed across additional facilities in Lineage’s network, the CTO (Jesse Crossno) commented "whelp, there goes the stealth mode". The "stealth" mode that CrossnoKaye tries to maintain is in my view counterproductive, given that an industrial control system cannot be just rolled out one day, it has to involve a lot of people over a long period of time with many careful iterations. 

                                                               * * *

I want to make it clear that the person who I notified about concerns for food quality degradation with respect to the potential energy saving procedure is a person of integrity and was known for voicing concerns to the management. That person is no longer with the company.

                                                               * * *

A fair question would be, if the VCs, who are not engineers, have been lead into thinking CrossnoKaye SaaS is a good fit for heavy industries, why did I (and other engineers) buy into the story, even ignoring the first warning flag that the founders of a technology company are non-technical? My answer is, there is an element of truth in their claims: the heavy industries do need to modernize their software for greater productivity, and there is an opportunity to reduce the energy consumption. Their story is also nicely packaged and hits all the right notes -- sustainability, saving people money, helping climate. The holes in that story started to appear early on but having had no experience in heavy industries, like literally everyone else who worked on software at CrossnoKaye, I thought I did not see the big picture. Eventually though, the big picture emerged and it was clear that their premise of a Saas-based "operating system" for heavy industries in the cloud was wrong; what I did not expect was that they would not want to discuss it. (The irony is that one of their stated principles, which they kept repeating in all-hands meetings, is "challenge assumptions" -- they even say it on their March 2023 "innovation summit" page.)

Most of all, however, I bought into the story because I was sure the people who gave this startup millions of dollars must have done their due diligence. 

Which, to be fair, they might have, to the degree they could since they are non-technical, seeing how they hired a law firm to investigate engineering problems. The blame may rest simply on the climate that put pressure on the VCs to invest in SaaS companies for 100x returns. As the Theranos whistleblower said, "I'm under pressure to exaggerate technology claims, exaggerate revenue projection claims. Sometimes investors will straight-up tell you, you need to double, quadruple, or 10x any revenue projection you think is realistic. I could see how this environment could create an Elizabeth Holmes." 

                                                               * * *

The pattern of half truths was there from the beginning. The recruiter who approached me before I applied in March of 2020 (different from the recruiter I mentioned above) said CrossnoKaye "have been able to bootstrap the entire project without VC funding and have generated over $5M in revenue this year", which sounds like profitability but -- to the degree it is even true -- turned out to be a one-off science project for which founders were hired separately from their company's product. As far as I can tell, the company is nowhere close to being profitable nor does it even have any clear path to profitability. 

Another thing the recruiter said in the initial email was that CrossnoKaye "is reducing energy consumption and costs from the world's biggest polluters by as much as 40%." I don't know if the 40% is repeatable across sites and not just another one-off under artificial conditions, but CrossnoKaye works only with cold storage facilities which are hardly "the world's biggest polluters". This claim, like their other ones, turned out to be all puffery and no substance.

                                                               * * *

All the information presented in this blog is either public, or, like the "millions of dollars" in funding, received outside of my employment with the company (the recruiter said in his email, "they are currently in the middle of a Series A raise targeting $20M at an $80M valuation"), or is my experience and memory interacting with people at CrossnoKaye, or is owned by me as is the case with my resignation letter which I sent from my personal email -- a legitimate means for expressing one party's will in writing for the purpose of ending a business contract -- and from which I redacted any information that could be considered proprietary or pertaining to the company's competitive advantage, per my employment contract. (I also redacted the names of people who were not at fault in any way.) 

Of course, I did not sign anything upon leaving the company. I did however exercise 10,000 shares, at 7c per share, from my employee stock options after I left.

                                                               * * *

Update 10/28/22: I learned that in early October one controls engineer who was a strong voice for safety, and who had been questioning the too-much-SaaS approach, was fired. I consider this engineer a person of integrity, so seeing that the management apparently decided to double down on eliminating dissent, I contacted members of the CrossnoKaye board who are part of the investment group that funded the company (IGSB) to let them know what was happening and pointed them to this blog. The response I received was that they are taking this issue very seriously and are investigating the matter.

                                                               * * *

Update 11/22/22: Erik Philipp, the COO, called me to tell me that my email to the Board has been shared with the company management, causing quite a stir he said, and that the Board had told them to hire an independent law firm to investigate the actions of the company. I was asked for permission to have my contact info shared with this law firm so they can meet with me, which I agreed to. I am not required to keep confidential any of the information I received, but, as a sign of appreciation for the lead investigator's work, I won't post updates until the investigation is completed.

                                                               * * *

Update 2/1/23: The investigation appears to be wrapping up. The law firm that investigated the company is Cooley LLP and the lead investigator who interviewed me on December 2nd is Randall Lee. Updates soon.

Meanwhile I have learned that at least four more employees have left on concerns similar to mine or been fired for voicing them. 

                                                               * * *

Update 3/12/23: Lineage Logistics, the international cold storage and food supply chain company mentioned above which has been using CrossoKaye software at pilot sites and which reported in December of 2022 that it aimed to deploy the software "at over twenty facilities in the next year alone" has been notified of this blog and of the investigation.

Ganging up

(posted Feb 5, 2023)

This is a record of one particular incident in May 2022 that shows how toxic was the environment I was in because of Adam and Perry (and to a lesser degree, Roman, who was my only peer at the team, lead by Adam and managed by Perry -- a peer behaving poorly is easy to handle when he doesn't have the team lead or the management support). 

This is written for CK people who are wondering if they are being gaslighted by Adam; read for yourself and decide. 

For everyone else, knowing this cast of characters may help you better understand the drama below:

• Roman: a software engineer in the controls team, which at the time consisted of him, myself, and Adam. Controls team writes software that controls the industrial system directly.
  
  
• Adam: the team lead for the controls team. When the company was smaller, Adam was the team lead for the entire software group, and when the software group was split into multiple teams, Adam was assigned to be merely the lead for this one team. On the day of announcement he looked quite bummed that his new status was lower than it used to be.
  
  
• Perry: the software team manager who is very nice and agreeable but who was completely unable to confront Drew -- and Adam, as you'll see below -- or really resolve any conflict. He had never acted disrespectfully towards me however, and even what I perceived as his parting shot ("it was nice knowing you") might have been a genuine goodbye.
  
  
• B****: a Science team guy with whom I worked closely on enabling sites and solving various problems, and have always had a very good work relationship with (I censored the names of people who are not at fault in any way)
  
  
• I**: an Ops (Operations) team girl with whom I also worked on several occasions on enabling different sites and with whom like with B**** I also enjoyed working. 
  
  
• Myself: the remaining member of the Controls team alongside Roman and Adam.

Here are the events that transpired as I recall them:

1.     End of May I prepare code which B**** and I** in the Ops tell me is needed to start reading W********** [facility] and which they say in the standup are waiting on for me. I invite Roman to review the code to make sure I did not miss anything safety wise but Roman blocks the code PR on aesthetic grounds; he first wrote a comment on the code in the company github page that something is very wrong, I replied that it was his own code block upon which he deletes his comment, but leaves the PR blocked

2.     I reach out to Roman in DM to clarify the issue and inform him that Ops are blocked on it; Roman claims code is "hard to read" and suggests a very different alternative that I believe is hard to read; I suggest since this is an aesthetic issue, if he agrees that the code works lets merge it to unblock Ops and then we can revisit later. Roman refuses and instead says "at least removed unused variables". The code in question was not unused variables but function parameters that I put for a specific reason. Roman insists they make the code "hard to read", falsely claims it fails the test (I prove that it doesn't), and speculates that "a new employee would be confused". I explain to him that every line of code I put in for a reason, that I was looking for help with correctness which he found no fault in, and I argue if the reviewer doesn't like the code he does not have to approve it but he should block the code only when it doesn't work, not when it doesn't meet his personal criteria of "hard to read"; nevertheless Roman refuses to lift the block still and demands I make the change to the style he prefers

3.     I say since we disagree, and I cannot move forward with his block in place, let's bring it up with the team, including the stakeholders, B**** and I** (Ops). Roman says he thinks stakeholders (Ops) should not be included, it is strictly a software question; I ignore that opinion and add the Ops to the meeting with Roman, Adam, Perry, and me.

4.     Adam immediately replies that my PR having changed 17 files and 170 lines is a "code smell".

5.     Perry jumps in and says Roman was correct to block my PR

6.     Roman defends his block on the grounds that my code has "poor readability", "bad design" and "no testing" (a previously unmentioned aspect that is outside of the scope, due to the nature of the code that was written long ago which his approach could not cover either).

7.     Moving from Slack to a video meeting between Roman, Adam, Perry and I, the three of them come on strongly that the PR should have been blocked, that the code I wrote is bad and needs to be changed, and we should do it since B**** and I** allowed for some extra time after Adam talked to them separately

8.     Perry admits at one point that the three of them are "ganging up on me" but says he believes the need for "software quality" justifies the ganging up.    

9.     I claim that the block was not done in good faith which is proved by the DM discussion between Roman and me in which I clarified all the (business) reason for the PR that were ignored, Roman's haggling ("at least remove unused vars") as well as Roman's deleted initial comment (for which he admits "I fucked up", but only in that video meeting, not in the PR where he made the comment or in the DM chat between all of us)

10.   Roman now backs off and says his blocking the PR was "miscommunication", implying he didn't have all the information (which is untrue, I spent over an hour in the initial DM to explain); Adam and Perry still justify the block

11.   Adam suggests a very different approach he calls "patterns" for solving the problem and in the interest of moving forward I agree to implement it.

12.   After several days when I implemented Adam's approach, in the best way possible, it turns out Adam's approach modifies 20 files and 200 lines, and is missing an opportunity for a new business need for the code that surfaced in the previous day; I also point out to the logical flaw in his approach that emerged clear

13.   In a standup Adam refuses to accept the argument and argues that the 20 files plus 200 lines is "not a code smell" because of potential for expansion (that doesn't exist). When I offer to write an "incident review" so that anyone in the software team can comment on he insists I shouldn't write it because I would be "biased", he said

14.   In a 1:1 with Perry, Perry says he now personally looked at my original PR and he agreed that it is the better solution and that I should check it in

15.   I ask Roman and Adam in a 3-way Slack chat if they agree with Perry's suggestion, because we as a team should arrive at the best code intellectually and not through an authority opinion. Roman responds that he, Perry and Adam met, unbeknownst to me, and decided that my original PR is better and should be approved; Adam does not chime in

16.   A couple of days later Adam sends a meeting invite to "catch up" saying there is no particular reason for this catching up, just wants to chat, and says "hey I haven't had time to chat with you, I've been working on building the leadership in this company". (Consistent with him being downcast on the day of announcement he would be merely a one small team lead instead of everyone's lead as he was in the first couple of years.)

17.   I turn the discussion to the issues we have, we touch on the PR and Adam says I had no reason to think there was anything wrong and in the same breath accounts for the time when W**** blocked his PR on unjustified grounds and how Adam was offended

18.   When I bring up our NPS score of * [a very low number] he says that me bringing it up "offends him". I insist on discussing it regardless, Adam claims the NPS score "would have been" high if only they had a Chromebook ready (which is not just speculative but also not true, our software wiped itself out).

19.   When I say how could this be the only reason our NPS score is very low when C**** is seriously pushing for a discussion on this issue, Adam says C**** has been "tasked" with that project by the management and that C**** is engaging in it merely because his performance will be evaluated on it -- implying lack of actual concern; this is very starkly at odds with my experience with C****

20.   Several times he attempted to explain away my reasoning about our problem by applying what I called psycho-babble -- "you feel that way because you are internalizing this and that", despite my stopping him every time he tried. After nearly 2 hours (meeting was scheduled for 15min) I leave with a strong impression that Adam is trying to manipulate me, compounded with his never admitting to being wrong

21.   In a 1:1 with Perry I raise that I have strong reasons to believe that Adam is exerting manipulative behavior towards me, I recount all the details from the previous meetings, and that there has been no discussion on what went wrong in our review process

22.   Perry denies that there are any such problems, says with Drew he had already received complaints by the time I complained (which if true raises the question why he hadn't acted) but with Adam he hasn't; I explain that as with Drew it takes time for manipulative behavior to be acknowledged (even J*** tried to defend Drew when he first reached out saying "I think he's a nice guy") but Perry dismisses this

23.   Perry defends Adam's "3 vs 1" approach -- in which Adam, after failing to produce a business reason why I should implement a sweeping change he proposed, said "there are three of us (Adam, Roman, and R******, although R****** was not fully on board) versus you" --  with "Adam had to decide". I say this is untrue, complete inaction on that front by anyone in subsequent  weeks and months shows there was no need to decide then, and that a developer cannot be coerced into making a change what he believes is wrong and detrimental to the business, especially of the kind that concerns nearly all stakeholders (ops, software, business, marketing, users, customers) and when the discussion at the company level has not been held and no valid argument in favor of the change has been produced.

24.   I tell Perry that I cannot join standups that Adam lead because of what I perceive as unfairness and attempts at manipulation and strongarming, that I do not want to put myself into stressful and unproductive situations, at least until what has happened is discussed; Perry does not say anything either way initially. Adam is away for a week during which I am present at standups; but when he is back I refuse to join, prompting Perry to DM to me the principles of Scrum, of which I am well aware, to which I urge him to bring the issue of what I see as incorrect and disrespectful behavior by my two team members, Roman and Adam. As nothing is resolved I tell Perry I am burned out by Adam’s disrespectful and manipulative behavior and Perry’s mishandling of the issue; he does not address any of it and occasionally sends a message or replies to my comments in Slack as if everything is normal

25.   In the remaining 2-3 weeks before the end of my employment I continue to work with other coworkers, particularly Ops, including nearly all day on the Juneteenth holiday to enable W********** [facility] reading data, and with M****, the new hire for the team with which I have a good working relationship and whom I help onboard; I continue monitoring the sites all day and on the weekends and helping troubleshoot issues, on one occasion because of these manual checks I catch a significant issue for which there was no alerts and N**** is very grateful for it, some in the software team comment that she should subscribe to "Davor as a Service" implying I provide valuable service not available otherwise; a week later the company attempts to terminate my employment.

As you see one teammate -- Roman -- tried to exercise power over me, I believe knowing that the team lead -- Adam -- will be on his side (since Roman had always taken Adam's side, as he did with Drew), and then both of them knowing that Perry will be compelled to do what Adam asks (Adam had claimed that he had told Perry, though Perry was his boss, to cancel the architecture discussion). 

This conflict with Roman is the event that eventually lead to the management trying to terminate my job, all because I would not allow a teammate to impose his will on me, as per Nassim Taleb's quote at the beginning of the blog. I have absolutely no regrets, and I very much recommend "speaking truth to power" and letting the chips fall where they may. 

Netflix model for ammonia control

(posted Apr 20, 2023)

I'll explain here why I think the "Netflix/SaaS" model for industrial control is disastrous.

With Netflix, the "brain" of the system is the software that runs in "the cloud" -- on "someone else's computer", on a giant farm of computers in Seattle, WA, or Ireland, or who knows where. The software orchestrates the experience for hundreds of millions of subscribers relying on a enormous database of movies and people who watch the movies and interact with each other. 

The only hardware you need besides your TV display is a device connected to the internet, maybe a Roku stick, that listens to what the brain in the cloud says and streams the pixels and audio to your TV. That device combined with your TV is like a dumb terminal for the mainframe.

The upsides are huge. To begin with, there is no DVD player you need to hook up to your TV and no DVDs to keep around. But more than that, you have a near infinite selection of movies available with a few clicks, the company can update the movies database in the cloud behind the scenes and add new navigation and search features in the cloud and on your locally running software. You can see what other people watch, get recommendations, rate, read reviews, comment and so on. Its a huge network of movie watchers consuming content and interacting through the cloud with each other by the hundreds of millions. You don't have to do anything to get all the latest and greatest stuff (except pay monthly for the software service). You can even stream through your Xbox or Nintendo Switch if for whatever reason you'd want to do it. This model is well suited here because the environment which consists of countless people who make and advertise and distribute movies and countless people who watch different movies (almost) every time and talk about the movies is, just like with music and video games, extremely dynamic. 

The downsides are small: if your internet is out you won't be able to watch a movie (and don't count on Netflix for DVDs come September). If the company releases faulty software by accident, either in the cloud, or in an update to your local hardware, you may not be able to log in or stream right then, but that's not a big deal, for obvious reasons; you'll go do something else until the service is back.

Now consider what happens when you apply the Netflix model to industrial cold storage facilities.

Today, these facilities typically have a Windows PC -- or several, so that if one crashes another takes over -- on which they run robust, time tested, battle hardened industrial control software. From these PCs network wires run into a network hub and from the hub more wires go into controllers that send electrical signals to giant compressors, blast fans, ammonia tanks and other hazardous equipment. When you press a button on the screen to open or close some valve, there is a reliable electrical connection between your PC and the end equipment which is often in the same building as you are.

(Often, control goes from the PC to a smaller, tighter, ultrareliable controller called a PLC -- programmable logic controller, which breaks down the PCs more complex requests into smaller ones, so instead of the PC talking to the equipment directly, the PC tells the PLC what to do and the PLC tells the equipment what to do.)

The control software is updated rarely and carefully; the electrical equipment is what it is, often years and decades old, and the refrigeration process by which you thaw or blast freezing air does not change much either -- food storage is a conservative enterprise.

This arrangement is not ideal: while the operators can log in remotely, typically via TeamViewer, it would be nice if they could do that from their phone in case they need to check the status of their equipment or initiate an action in an emergency when they are not next to their PC. More than that, you would like to be able to optimize the way these machines, which consume huge amounts of electricity, operate, based on your business patterns, and thus save energy and reduce costs to be more competitive. Of course that comes far distant second to the safety of the facility people and the food. (You don't want to damage the food either by fluctuating the freeze temperatures more than a couple of degrees.)

This is the "DVD player" model of the facility: 

(what looks like a monkey stealing a rocket is a symbol for a piece of equipment called a "compressor").

With the Netflix model, that Windows PC is taken out of the equation, or rather relegated to being used only as a browser. There is an advantage to it: you don't have to maintain the PC, a Chromebook fresh out of the box can do the job (as your phone or tablet can). All you need is a browser; you can throw your "DVD collection" along with the "player" in the trash.

Of course when you use only the browser, there still needs to be some piece of hardware -- the equivalent of a Roku stick -- placed in the facility to talk to the equipment but you do not control it anymore: it receives its orders from the cloud. What is worse, you have absolutely no control over which software runs in the cloud. Whereas previously you knew that the software is entirely contained within your PC, and you had that direct connection from the PC to the equipment nearby, now there are hundreds and thousands of components on which hundreds and thousands of people -- most of them completely unknown to you, working for companies you have never heard of -- are constantly updating, changing, reversing, and doing who knows what to the software that is the lifeblood of your facility.

(And if there was a PLC between the PC and the equipment, the PLC now has to forward equipment commands from the program defined in the cloud onto the equipment. Everyone has to submit to the cloud.)

This is the "Cloud Controlled" model.

The "mainframe" -- the "brain" and the database -- is now in the cloud. Sometimes you literally cannot even log in to turn off that valve that may be in your eyesight because something called "Okta" has failed somewhere "in the cloud". (This happened, several times.) Or, something called "github", also in the cloud, can't be reached and your system cannot boot. With the cloud in the critical path, you may easily end up locked out of being able to control hazardous equipment in the room next to you, and the only thing you can do is call support. 

(Which, has to be said, was great, very responsive, day and night, but you would much rather have essential things work like they used to and not have to call someone at 11pm. And support can't do much if Okta or any number of 3rd party components needed for the cloud system to work is experiencing outages.)

Or, simply, your internet is out. You have been given an alternative to connect to the "Roku stick" the company has installed in your facility directly in case of emergency, but that is not the default way because it has limited functionality: the Roku stick is tiny. And unless you use this approach all the time, an outage disrupts your flow at best and makes you vulnerable in an emergency at worst, when it may be dark and hazard lights are flashing and you have to remember how to use this alternative way. (Also happened.)

With the cloud, you depend on countless third party providers who depend on yet other providers and you have no control over any of it. And this in facilities that often even have their own electricity should the power grid fail.

More dangerous still is that some software guy in the vendor company who has no idea what you are doing and what your business needs are at the moment has decided to "push" a software update because he decided some cool new feature needs to be in, and your system stops working (also happened), or is doing something differently than it used to (also happened, and of course you don't notice it immediately, sometimes it takes days). Yes, the support notifies the software people promptly, but how easy is it to figure out at midnight what exactly caused the bug? And, unlike with the PC software, in a complex cloud system you often cannot restore the system to the previous working state (also happened). You have to find the bug and fix it while time is running out.

To top it off, because of the Netflix cloud model, adding one more "movie watcher" -- another facility -- to the single shared database can wreck the database and/or the "brain" in the cloud and cause the software to misbehave in infinite and unpredictable ways -- goes offline, or stops running, or turns off all the alarms. But unlike Netflix, even when everything works great you get zero benefits from this sharing. You don't care what another cold storage facility 500 miles away (or 50 miles away) is doing, you are not going to rate a software feature and review other ratings by other facilities.

That is the "completely unnecessary, architecturally wrong interdependence of facilities and overdependence on the cloud" mentioned earlier in the blog. Even your Office 365 subscription allows you to use Word on your PC independently of the Microsoft cloud (unless you want a fancy new template). You update your Word when you want.

The Netflix model is like a beehive; it opens up the software so that countless people can keep constructing and deconstructing it while countless other people feed it with data and consume from it:

(source: goodtechthings.com) 

This beehive nature of the model which allows the software to change rapidly at the cost of instabilities and failures absorbed by individual "bees" is incompatible with industrial control where processes require stability. Midnight heroics happen when software people prevent disasters made by other (or same!) software people who messed things up in the perpetual quest for new features. But disaster at Netflix is very different from a disaster in an industrial facility.

This does not mean facilities should use their Windows 7 PCs forever and without ever going online; progress is inevitable, and there is certainly room for collecting facility equipment data and sending it to the cloud for people and machines to analyze it to optimize the business, as long as it doesn't endanger the essential function. There is also room for getting limited feedback from the cloud, in which the cloud, optionally, sends suggestions for optimization strategies that the local software can apply to save energy. But the integrity and independence of the local facility must be respected at all times.

This would be the "Cloud Assisted" model:

(The images were made on my home computer and outside of working hours. They merely describe the obvious architectural choices.)

The problem with this alternative approach is, as far as investors are concerned, it's not that much "SaaS" anymore; SaaS becomes a sideshow. You still have to have considerable hardware on the premises -- a powerful PC or multiple ones for failover -- and the company can't update its software whenever it wants; its business doesn't "scale" as much, which investors don't like, even if it could provide valuable service to the customer, because everything is slowed down a lot more. 

Except there are cases where slow is good, and this is one of them: industrial cold storage, not to mention food processing, is a dangerous activity. You have to roll out features conservatively and test extensively. There is no other way. "Move fast and break things" will break things.

"Cloud-assisted" is just one alternative approach to "Cloud-controlled", there are others and certainly better ones, like the local cloud -- an on-premises cloud. But whatever the more appropriate solution, anything other than the "cloud-controlled" approach was never discussed.

(All this doesn't take into account security, which is a topic I do not have expertise in.)

* * *

Less obvious and maybe the most concerning aspect of this kind of scheme is, it would take the power away from the facility people and put it in the hands of the software people outside of your company. To save some money -- maybe -- you give those software people the keys to your kingdom. And that's the best case scenario.

My opinion is, if companies are going to use a software service in a way that affects their livelihood, not to mention life and limb, they should make sure that control remains in their hands at all times. 

Restraining Order

On March 29, 2023, CrossnoKaye filed a 120-page petition for restraining order against me, listing this blog as the primary evidence and citing fears for their safety and "safety of their investors and clients". At the preliminary hearing on April 24, 2023, I requested the case to be presided by a Judge, and the request was granted. On May 10, 2023, at the California Superior Court of Santa Barbara, the hearing was held on the matter of the restraining order for the case of "CrossnoKaye vs. Davor Magdic". 

The company lawyers (they had two; I represented myself) won the stay for the "classic" part of the restraining order -- the main part of which is the prohibition from me being within 100 yards of the company offices and four of its employees (three of which are executives), which I have neither attempted nor have had any interest in whatsoever since I left, but for which they argued using this blog, parts of messages taken out of context, and the Glassdoor review. The gag order the company requested was denied: the Court upheld my right to tell investors and clients about the company's practices.

The Company had requested that I be ordered to not "Contact the clients or investors of CrossnoKaye, Inc., either directly or indirectly, by any means, including, but not limited to, in person, in writing, by public or private mail, by interoffice mail, by text message, by fax, or by other electronic means." In the supporting material, the Company tried to justify this request by alleging fears for the safety of their investors and clients without providing any evidence to support such concerns. Furthermore, the nature of the restraining order petition requires that those investors and clients who are to be protected are listed by their names; not only did the Company not provide the names, it would have been impossible to do so (Lineage Logistics alone has an estimated 17,000 employees).

Yet the Commissioner who approved the Temporary order, while not granting the gag order, wrote that "the issue of communication with investors and clients will be addressed at the hearing", despite the lack of grounds for considering such a ban on communication. The only piece of communication with investors and clients the Company provided was my up to that point the only letter to the investors, which contradicted their claims as the letter was respectful, mindful of the sensitivity of the issue, and even suggested promoting one of the persons for whom the Company sought protection; the Company presumably included it to support the claim of an "escalating" pattern of communication. They did not include copies of my correspondence with the client, whether it was available to them or not, and which was equally respectful.

Not being able to go near the Company offices or the employees had no impact on me as I had no interest nor have ever made any attempt to do so since I left, but the gag order attempt was a direct attack on my First Amendment rights. It was broad enough that, had it been granted, it could and probably would have been used to try to shut down this blog. For those reasons, after I submitted my written response, at the preliminary hearing with the Commissioner I requested the the case be heard by a Judge, and the request was granted.

On April 27, 2024, three days after the preliminary hearing, I sent an email to a CrossnoKaye investor to invite him to attend the hearing on May 10 and see for himself what the Company is doing. This investor has in the past emphasized the importance of building not just startups, but startups that are a valuable part of the community. One of the employees I spoke to expressed that, after Crossno Kaye, she felt that "this entire startup scene is a lie."

At the hearing, where the Judge confirmed that he had read my written response in full, the Company did not even try to argue in favor of the gag order; and while in their petition they tried to establish that I am disgruntled by claiming they terminated me -- despite citing the name of my blog, "How I Quit My Job At CrossnoKaye", numerous times in the petition -- now that the full correspondence following my resignation was revealed to the Court, they referred to my departure only as me being "separated" from the Company. 

The Company also argued that the blog paragraph, "If everyone is respectful and I disagree with the company direction, I wish them good luck and leave. But if someone in a position of power tries to disrespect me repeatedly and the management allows it, I don't leave" means I am saying I refused to leave the company. This, in my opinion, makes no sense on multiple levels, and neither does the premise that someone who makes a considerable effort to call to public attention the actions of a company and his own would threaten someone's safety -- in the same way that a company allegedly concerned for the safety of four employees would shift focus towards asking for a complete ban on contact with all of its clients and investors. 

Regardless, the Court decided as it did, and I respect its decision.

                                              * * *

The case can be found at https://portal.sbcourts.org/CASBPORTAL, case number "23CV01355".

                                              * * *

This is the email I sent to the investor on April 27, 2023:

Dear Mr. Duca,

I am a former CrossnoKaye software engineer and a shareholder who reached out to IGSB last year, via your former colleague Mr. **** ****, to notify IGSB of ethics and safety concerns in CrossnoKaye.

I am writing to let you know that on April 4th this year CrossnoKaye filed a petition with the Santa Barbara Court for a “workplace violence” restraining order against me in what is clearly an attempt at legal intimidation. Alleging fears for the safety of "employees, investors, and clients", CrossnoKaye requested protection for four employees (three executives and a non-executive), and the company also requested a complete ban on contact of any kind with investors and clients. The first request, as is often the case, was temporarily granted; the second was not. That this second request was denied by the Court upholds my right to tell you about this petition, and that the request was made in the first place indicates CrossnoKaye may not want you to know about it. 

To support groundless claims of a credible threat of violence, the Petitioner goes as far as to cite a literary quote that opens my blog "How I Quit My Job At CrossnoKaye" as a reason to fear for the safety of not just the four employees but the "investors and clients with whom I share those workspaces" as one declaration states, heinously implying that I might carry out an act of violence at the company premises and implausibly using this alleged concern to justify asking for a gag order on communicating “directly or indirectly” with tens of thousands of people across the country who count among the company investors and clients. To establish an "escalating pattern" of threats the company even claims in its filing that the quote was added recently, which evidence proves is both false and known to the company to be false. I have provided a detailed response to the Petitioner and the Court, and, at the hearing last Monday, April 24, I requested that the case is heard not by the Commissioner, who partly approved the temporary restraining order, but by a Judge. The request was granted, and the hearing was rescheduled for May 10.

I invite you to attend the hearing if you are at all available to do so. The hearing will take place in the Superior Court of California, county of Santa Barbara, Department 3, on May 10 at 1:30pm and will be presided by Judge Thomas P. Anderle. My party overheard CrossnoKaye’s two attorneys that they would ask that the hearing be confidential, but I assume you will be granted access if you choose to attend.

If you need any information from me, please feel welcome to reach out on email or phone at any time.

I did not receive a response, and the investor was not present at the hearing. 

                                                   ***

At the preliminary hearing, after I apparently surprised everyone by requesting a hearing by the Judge, the Company party (all six of them) and me and my party (my girlfriend who came to testify that a claim in their evidence was false, and made false knowingly) left the courtroom. My girlfriend and I went back to my car, where I realized I forgot my laptop bag in the courtroom so we went back. Arriving back to the court building some 5 minutes later, we saw the Company party standing outside and huddled in an intense conversation. Clearly they did not expect to see us. We got inside and through security, collected my laptop bag from the courtroom where different case was being heard, and exited the building. By then the Company party had left, but as we went back towards the car, we saw them at the nearest street corner, still very animated. We went around the block to not infringe on the temporary order. 10 minutes later, we observed them from a safe distance, still standing there and discussing. It appeared that the randomly assigned Judge Anderle, known for his conservative views towards freedom of speech, was not favorable to their cause.

The obvious question was why the matter required so much legal firepower and concern. One look at the form for Restraining Order for "workplace violence" -- I had the opportunity to research the matter while writing a response -- makes it obvious that the form was meant to be filed without an attorney, to quickly give the police a green light to stop a crazy person from causing harm, and is almost always filed the next day after an alarming event, not weeks and months after. It is also far more common that the Respondent -- the person against whom the restraining order is sought -- needs an attorney, not the Petitioner. CrossnoKaye brought on two.

                                                   ***

I will post here the relevant documents from the hearing once I determine what can be published without breaking compliance with the order.