This is just the beginning for sharing metrics like this. We're looking forward to feedback on this batch of results which will help form other analyses that would be interesting to see about the dev process in general.
We believe there's a lot of information locked in dev tools like code review that can shed quantitative light on how those processes lead to successes or failures in development. This is in partnership with the kind of anecdototes and best practices experienced devs and managers already frequently share.
Nick here, CEO/Founder of AutoDevTech. The team and I are proud to release AutoDev Analytics: reports and benchmarks measuring dev processes based on thousands of big, popular projects on GitHub. We want to share what we’ve learned about the dev process and code review, get feedback, and open up free analytics that compare your team against thousands of other code bases.
We’ve got stats showing most code doesn’t get any feedback in code review[1], unreviewed code is twice as likely to churn out over a year than reviewed code[2], and teams spend about 3 hours per week reviewing code[3]. And we’ve got benchmarks for dozens of other metrics like code review turnaround and monthly codebase churn.
Take a look, sign up for analytics and let us know what you think.
Would love to see a little more on the landing page than just one screenshot. How about more description of features, why this is better than comparable options paid and free. How about a live version we can see in action? What does deployment of an agent look like?
Come join the team as one of our first engineers building tools to help developers write better code. We’ve got a dataset of code and outcomes like review feedback and bugs and we’re using that to train ML models to predict what happens to code as developers build new features.
We’re looking for versatile developers interested in an early-stage startup experience building user experiences to assist devs across the development lifecycle (think code reviews, testing, fixing bugs). There are a lot of challenges presenting information like this in web apps and 3rd party integrations. We’re looking for someone to help define and solve those challenges.
The team consists of industry vets, led by an engineering and product head with multiple startup successes from founding to exit. We’re looking to grow a supportive, challenging, fast-paced culture focused on building some of the best technology and products in the industry.
Fascinating video. It's cool to see a collaboration of child development and reinforcement learning in action and to hear a research speak about both in an experimental context.
The comment in a different thread about Diamond Age comes to mind, and here we see some of those elements playing out.
Nod. I wonder what might be done with richer input, gaze and pose, rather than just facial affect (and very limited interaction context).[1] AR seems likely to make those much more widely available - though it could be worked on now, but for lack of institutional context.
On the other hand... there are other pressing bottlenecks to a Primer. Status quo has people learning the Sun is yellow, from Kindergarten teachers, college textbooks, and even outreach. With only a very very few of them getting an "oops, nope, our bad" correction years later in grad school, discussing common misconceptions in astronomy education. We're collectively not able, on a timescale of decades, to even get Sun color right in the most popular college textbooks. So if a Primer is to teach science and engineering as a richly interwoven coherent tapestry... we'll need to figure out socially how to transformatively improve our creation of those stories.
The student in the video is learning the color lavender. Learning color is common preK-6. And yet, even first-tier physical-sciences graduate students are deeply steeped in misconceptions about color. So a motivational peer robot might help... at least with equity. But if we aspire to teach color successfully, a novel goal, attention seems needed elsewhere. But there aren't great incentives to ask "What might a greenfield tech-enabled preK-6 successful approach to teaching color look like?" Or at least, I've not seen it. If anyone knows of a setting for such questions, I'd greatly appreciate hearing of it.
I think this is a reasonable question. Usage based pricing seems like a trending topic lately and I hear Slack called out as a good example. Monthly seat licenses were a practice that existed long before more obvious "usage based pricing" models became prominent.
See the comment from @jasoncwarner about GitHub actions being a platform for much more than CI.
I wonder how far that extends to non-GitHub provided services. For instance, could we leverage GitHub actions, perhaps even Flat Data, to scrape some web site and store it (perhaps uploading elsewhere) in a more comprehensive way vs. storing some small snippet of the data in a git repo?
Yes. Or S3 bucket, or whatever. The thing I'm getting at is, can we use GitHub actions for application tasks like web sraping that need compute and network access, but that don't really do much with with a git repo. Does GitHub want to support that?
I don't know much about Flat Data, but I'm impressed with how much GitHub is doing as GitHub since the MSFT acquisition. They continue to offer compelling services to developers, and increasingly to enterprise customers. All without abandoning much of what made GitHub great: a focus on developers and easy to access dev productivity.
Notice the prominence of the VSCode integration here. Notice the dramatically increased presence of MSFT on GitHub in general. It seems like they've managed to integrate these two cultures and product-sets in sensible ways. Given how hard big integrations like this are to pull off, I feel like the community really dodged a bullet in terms of access to products/tools.
The OCTO DevEx team reaaaaaallly loves VS Code — beyond the editor, it's just a great surface for experimental developer tooling!
GitHub Codespaces aren't generally available yet, but being able to target both "native" VS Code as well as in-browser VS Code with the same extension is super powerful. Expect a lot more from us on that front.
We've also released a pair of little projects re VS Code development that we've extracted from our work:
https://github.com/githubocto/tailwind-vscode: a Tailwind CSS plugin which creates Tailwind color tokens for each of the VS Code theme colors, easing theme-native styling in VS Code.
Honestly, I wish I was able to get into the Codespace Preview so I could play around with a lot the experimental features it offers - Especially the tie in with Docker Desktop Dev Environments, my whole development workflow is likely going to change drastically this year.
Agree, it's important that we keep an eye on things and, however we can, hold MSFT and GitHub accountable to keep up the good showing.
We've seen new features launched (e.g. this one) long enough after the acquisition that much (most, all?) of the work happened in the post acquisition environment that I'm optimistic. But I've been wrong before.
It's already here, is just that the userbase and third parties are (happily) doing the dirty work for them. Try going GitHub-free for a month or three and you'll notice how many things rest on the assumption that you have a GitHub account. "Log in with GitHub" is essentially what Microsoft hoped for with Passport, if Passport had actually been successful.
Look at how it shat on Markdown with what it calls "GitHub Flavored Markdown". Look at the things that it calls "wikis". Look at how GitHub's PR merge tool junks up the commit log. Look at how many projects don't even have a way to accept a fix unless you submit it with GitHub's janky pull request workflow. Hell, a bug in Netlify's command-line client managed to make its way into release versions that would straight up cause the process to terminate when cwd was a repo that wasn't hosted on github.com, leading to unhandled exception.
The tacit assumption that you're using GitHub is like the tacit assumption 15 years ago that you were using Visual Studio, only this time, you can't escape just by steering clear of Windows-related tech.
GitHub Flavored Markdown seems like a nice extension to Markdown to me. Fenced code blocks? Great idea. Lots of other flavors of Markdown do the same thing. I don't know who's the leader or follower here, but I'm glad they're doing it. I'm not sure what's the gold standard for wikis, but they all seem like the kind of thing every vendor has similar, flawed, good-enough solutions for. And I know there are other thoughts around how to manage merges, but having a merge commit (or a squash merge or a fast forward) seems like a reasonable contender for handling a feature branch. But maybe there's something I'm missing? I guess any hegemony is bad for innovation?
Are there a lot of walled gardens that only allow sign-in with GitHub? That's not really an issue I've run into. I can't think of any site I want to invite my aunt/uncle/cousin to log into that only accepts GitHub login. In fact, I'm not sure there's a lot of tools I want my colleagues to use that require a GitHub login that isn't already tied to a GitHub hosted repo.
They acquired a company that was doing the thing that they are wont to do and are criticized for, and have poured the significant resources at their disposal into growing the circle of impact. Where it originates from and whether it was or wasn't already independently in full swing (or partial, in this case) before their involvement doesn't matter, the effect on the user is the same. Besides that, if a person's problem with a given practice is whether or not Microsoft is the perpetrator, then that person is a hypocrite and doesn't actually give a shit about the the thing they claim to be concerned about.
The GitHub acquisition likely the single catalyst that showed me Microsoft has actually pivoted how they're approaching business, and are at least putting good faith efforts to do better.
With literally 0 actual knowledge to reference on how GitHub has felt since then, outwardly it feels like MSFT has played to all of their strengths (money, infrastructure, money, "developers, "developers", "developers" memes) and amplified what GitHub had been pushing for.
I only hope it stays going well, because it's certainly keeping me engaged in trying more MSFT products and services than I likely would've otherwise even glanced at.
AutoDevTech | Developers, Data Scientists | Full-time | Seattle/Remote US
I’m building AI/ML to help devs write better code by giving customized, actionable feedback learned from existing coding patterns and outcome indicators like review feedback, bugs and production data. I’ve got funding, a very large and growing dataset, labels and a ton of technical challenges to solve.
I’m hiring the initial team: data scientists in ML/NLP, topic modelling, classification of semi-structured data, plus developers that can build UI/UX, 3rd party integrations and support large-scale data pipelines. I want people looking to learn a ton and launch a product from scratch.
We’re going to be part of a big change in how computing supports the creative process throughout software development.
AutoDevTech | Developers, Data Scientists | Full-time | Seattle/Remote
We’re building AI/ML to fundamentally change the way devs write code: giving customized, actionable feedback, and researching bugs by learning from existing coding patterns.
We’ve got funding to assemble the early team and are looking for motivated, versatile developers and data scientists that want to build the initial tech/product at an early-stage startup.
Contact careers@autodevtech.com for more information.
We believe there's a lot of information locked in dev tools like code review that can shed quantitative light on how those processes lead to successes or failures in development. This is in partnership with the kind of anecdototes and best practices experienced devs and managers already frequently share.