1  ---
 2  name: Roadmap issue
 3  about: Use this template for tracking release roadmaps.
 4  title: "Roadmap for vXXXXX"
 5  labels: roadmap
 6  assignees: zimmski
 7  ---
 8  
 9  Tasks/Goals:
10  
11  - [ ] Development & Management 🛠️
12    - [ ] TODO what and why as goal
13  - [ ] Documentation 📚
14    - [ ] TODO what and why as goal
15  - [ ] Evaluation ⏱️
16    - [ ] TODO what and why as goal
17  - [ ] Models 🤖
18    - [ ] TODO what and why as goal
19  - [ ] Reports & Metrics 🗒️
20    - [ ] TODO what and why as goal
21  - [ ] Operating Systems 🖥️
22    - [ ] TODO what and why as goal
23  - [ ] Tools 🧰
24    - [ ] TODO what and why as goal
25  - [ ] Tasks 🔢
26    - [ ] TODO what and why as goal
27  - [ ] Closed PR / not-implemented issue 🚫
28    - [ ] TODO what and why with reason
29  
30  Release version of this roadmap issue:
31  
32  > ❓ When should a release happen? Check the [`README`](../../README.md#when-and-how-to-release)!
33  
34  - [ ] Do a full evaluation with the version
35    - [ ] Exclude certain Openrouter models by default
36      - [ ] `nitro` cause they are just faster
37      - [ ] `extended` cause longer context windows don't matter for our tasks
38      - [ ] `free` and `auto` cause these are just "aliases" for existing models
39    - [ ] Exclude special-purpose models
40      - [ ] Vision models
41      - [ ] Roleplay and creative writing models
42      - [ ] Classification models
43      - [ ] Models with internet access (usually denoted by `-online` suffix)
44      - [ ] Models with extended context windows (usually denoted by `-1234K` suffix)
45    - [ ] Always prefer fine tuned (`-instruct`, `-chat`) models over a plain base model
46  - [ ] Tag version (tag can be moved in case important merges happen afterwards)
47  - [ ] For all issues of the current milestone, one by one, add them to the roadmap tasks (it is ok if a task has multiple issues) with the users that worked on it
48    - Fixed bugs should always be sorted into respective relevant categories and not in a generic "Bugs" category!
49  - [ ] For all PRs of the current milestone, one by one, add them to the roadmap tasks (it is ok if a task has multiple issues) with the users that worked on it
50    - Fixed bugs should always be sorted into respective relevant categories and not in a generic "Bugs" category!
51  - [ ] Search all issues for ...
52    - [ ] Unassigned issues that are closed, and assign them someone
53    - [ ] Issues without a milestone, and assign them a milestone
54    - [ ] Issues without a label, and assign them at least one label
55  - [ ] Write the release notes:
56    - [ ] Use the tasks that are already there for the release note outline
57    - [ ] Add highlighted features based on the done tasks, sort by how many users would use the feature
58  - [ ] Do the release
59    - [ ] With the release notes
60    - [ ] Set as latest release
61  - [ ] Prepare the next roadmap
62    - [ ] Create a milestone for the next release
63    - [ ] Create a new roadmap issue for the next release
64      - [ ] Move all open tasks/TODOs from this roadmap issue to the next roadmap issue.
65      - [ ] Move every comment of this roadmap issue as a TODO to the next roadmap issue. Mark when done with a :rocket: emoji.
66  - [ ] Blog post containing evaluation results, new features and learnings
67    - [ ] Update README with blog post link and new header image
68    - [ ] Update repository link with blog post link
69    - [ ] https://github.com/symflower/eval-dev-quality/discussions
70      - [ ] Remove the previous announcements
71      - [ ] Add a "Deep dive: $blog-post-title" announcement for the blog post
72      - [ ] Add a "v$version: $summary-of-highlights" announcement for the release
73  - [ ] Announce release
74  - [ ] Eat cake 🎂
75  
76  TODO sort and sort out:
77  
78  - [ ] TODO