ocarina/design.txt

966 lines
27 KiB
Plaintext
Raw Normal View History

===============================================================================
= =
= Ocarina 6.0 =
= =
===============================================================================
My main goal for Ocarina 6.x is to plan out all of my actions before writing
code. In the past I was adding features as I thought of them before thinking
out how everything works together, and this made Ocarina difficult to maintain
because I had no overall plan. This document aims to fix that.
I will also create unit tests as I add features so bugs can be found faster.
Unit tests will be created for each module (file) in my backend library code.
Files:
$HOME/.ocarina{-debug}/
album.db
artist.db
genre.db
groups.idx
library.db
playlists.lst
track.db
/usr/bin/
ocarina
/usr/lib/ocarina/
ocarina/
design.txt
ocarina/gui/
*
ocarina/include/
audio.h
database.h
database.hpp
file.h
filter.h
groups.h
idle.h
idle.hpp
library.h
playlist.h
print.h
version.h
ocarina/lib/
audio.cpp
database.cpp
file.cpp
filter.cpp
groups.cpp
idle.cpp
library.cpp
playlist.cpp
Install:
Ocarina will be compiled into a single executable placed under
/usr/bin/. Any extra files needed to run will be placed under
/usr/lib/ocarina/.
Printing: (include/print.h>
Sometimes text needs to be printed to the screen so users (or debuggers)
know what is going on. Enabling dprint() when CONFIG_TEST is enabled
means I will only need a single test.good file for output comparing.
API:
void print(string fmt, ...)
Print text to the screen.
void dprint(string fmt, ...)
Print text to the screen when debugging or testing is enabled.
Versioning: (include/version.h)
This file contains a simple function for returning a string stating
the current version.
API:
const char *get_version();
Returns a string describing the current version.
On-disk files: (lib/file.cpp)
Data will be stored in the user's home directory according to the
XDG / freedesktop.org specification. This means storing data in
$XDG_DATA_HOME/ocarina{-debug|-test}/ and storing configuration in
$XDG_CONFIG_HOME/ocarina{-debug|-test}/. In addition, I will support
importing data from Ocarina 5.10 for conversion to the new format.
In theory my file format will not change often, so it should be
possible to use multiple Ocarina versions with the same data. However,
should the format need to change I will only support forward
compatibility. This means that downgrades will not be possible after
a file format change. To keep the code even cleaner, I will only
support updating from the previous file format version. This means
that legacy support will be removed after the first file format change.
Items should be written to a file with either a space or new line
separating multiple values.
- Notation:
File << aaaaa << bbbbb << endl is translated into "aaaaa bbbbb\n"
- File version:
#define FILE_VERSION 0
- Hint where the file is located:
enum FileLocHint {
FILE_TYPE_CONFIG,
FILE_TYPE_DATA,
FILE_TYPE_LEGACY,
FILE_TYPE_INVALID,
}
- Open mode:
enum OpenMode {
OPEN_READ,
OPEN_WRITE,
NOT_OPEN,
}
- File:
class File : std::fstream {
private:
unsigned int version;
OpenMode mode;
FileLocHint hint;
string filepath;
public:
File(string, FileLocHint);
~File();
const char *get_filepath();
const unsigned int get_version();
bool exists();
bool open(OpenMode);
bool close();
string getline();
}
- File format:
File << FILE_VERSION << endl;
File << <OTHER_DATA>;
- API:
File :: File(string filepath, FileLocHint hint)
Resolve filepath to one of:
XDG_{CONFIG|DATA}_HOME/ocarina/filepath
XDG_{CONFIG|DATA}_HOME/ocarina-debug/filepath
XDG_{CONFIG|DATA}_HOME/ocarina-test/filepath
$HOME/.ocarina/
$HOME/.ocarina-debug/
If filepath is an empty string, set the file hint to
FILE_TYPE_INVALID and do not set the filepath field.
File :: ~File()
Close the file stream if it is open.
const char *File :: get_filepath()
Return the full filepath to the file.
const unsigned int File :: get_version()
Return the file version number.
bool File :: exists()
Return true if the file exists in the filesystem.
Return false otherwise.
bool File :: open(OpenMode mode)
When opening a file for reading (mode == OPEN_READ),
- Return false if the file does not exist
- Open the file
- Read in version from the start of the file
- Return true
When opening a file for writing (mode == OPEN_WRITE),
- Return false if the file has FILE_TYPE_LEGACY set
- Create missing directories as needed
- Write version information to the start of the file
- Return true
Return false if hint == FILE_TYPE_INVALID.
Return false if the file is already open.
Return false if there are any other errors.
bool File :: close()
Close a file after IO.
string File :: getline();
Read an entire line from the file and return it to the caller.
In theory a return value optimization will occur so returning
a string by value won't be a problem.
Database: (lib/database.cpp)
Ocarina 5.x created a different save file format for each type of
data that needed to be stored (preferences, library paths, playlists).
I intend to unify everything into a generic file format that can be
accessed through a generic database interface. The database code will
be in charge of printing the "valid" bit for each DatabaseEntry so that
child classes do not need to call into the parent class. If valid ==
true, the DatabaseEntry will be streamed out followed by a newline. If
valid == false the database will print the next entry in the vector.
Modules should inherit from the DatabasEntry class and implement their
own read() and write() functions. The "valid" field will be stored
before these functions are called, and the entry will be skipped if
valid is set to false.
The Database class is a templated class, so code could potentially
get messy. Normal class declarations can still exist in the file
include/database.h and member functions can be written in the file
include/database.hpp, which will be included by database.h. Any
function not relying on a template can be written in lib/database.cpp.
- DatabaseEntry:
class DatabaseEntry { /* let database modify valid flag */
public:
bool valid;
virtual const std::string &primary_key() = 0;
virtual void write(File &) = 0;
virtual void read(File &) = 0;
virtual void print() = 0;
};
File << <CHILD_CLASS_DATA>
- IndexEntry:
class IndexEntry : public DatabaseEntry {
public:
string key;
vector<unsigned int> values;
const std::string &primary_key();
void write(File &);
void read(File &);
void print();
};
File << key << endl;
File << values.size() << values[0] << .. << values[N] << endl;
- Database:
template <class T>
class Database {
private:
vector<T> db;
map<std::string, unsigned int> keys;
unsigned int _size; /* Number of valid rows */
File file;
public:
Database::Database(filename, flags);
void load();
void save();
void clear();
void print();
unsigned int insert(T);
void delete(unsigned int);
unsigned int size();
unsigned int num_rows();
unsigned int first();
unsigned int last();
unsigned int next(unsigned int &);
unsigned int find(const std::string &);
T &operator[](unsigned int);
};
File << db.size() << endl
File << INDEX_0 << db[INDEX_0].valid << db[INDEX_0] << endl;
File << INDEX_1 << db[INDEX_1].valid << db[INDEX_1] << endl;
...
- API:
Database :: Database(filename, flags);
Initializes database to use ~/.ocarina{-debug}/filename.
Set up flags.
void Database :: load();
Reads a saved database from disk.
void Database :: save();
Saves the database to disk.
void Database :: clear();
This function exists only if CONFIG_DEBUG is enabled.
Clear the database contents in-memory, but do NOT write
to disk.
void Database :: print()
This function exists only If CONFIG_DEBUG is enabled.
Following a similar format for writing to disk, print the
database to the console in a human-readable format.
template <class T>
unsigned int Database :: insert(T &);
Adds a new item to the db and marks it as valid. A reverse
mapping is also created into the keys map to map the primary
key back to the id of the newly added item.
void Database :: delete(unsigned int index);
Mark db[index] as invalid (quick deletion).
unsigned int Database :: size();
Returns number of valid rows in the database.
unsigned int Database :: num_rows();
Return db.size().
unsigned int Database :: first();
Return the id to the first valid row.
unsigned int Database :: last();
Return the id of the last valid row.
unsigned int Database :: next(unsigned int &id)
Return the id of the next valid row or return db.size()
if there are no remaining valid rows.
template <class T>
T &Database :: find(const std::string &key);
Search for primary key "key" in the database. The reverse
mapping should be used to make this operation faster. Throw
an empty exception if the key is not found in the mapping.
template <class T>
T &Database :: operator[unsigned int index]
Return a reference to db[index]. If index is out of range,
throw an empty exception.
Filter: (lib/filter.cpp)
Filtering is used to generate a subset of songs displayed by the UI to
that users can choose from. The inverted index is generated at startup
so there is no need for a remove() function, since it will be wiped
the next time the application starts.
- Index:
Database<database :: IndexEntry> filter_index;
map<string, string> lowercase_cache;
unsigned int lowercase_cache_hits;
- Parsing:
1) Convert the provided string into a list of words, using whitespace
and the following characters as delimiters: \/,;()_-~+"
For each word:
2) Check the lowercase_cache to see if we have seen the word before,
a) If we have, return the stored string
b) Convert the string to lowercase and strip out remaining
special characters. Add the result to the lowercase_cache;
- API:
void filter :: add(string, track_id);
Parse the string into substrings following the "Parsing"
section (above). Add each (substring, track_id) pair to the
filter_index.
To generate substrings, iterate over the word starting from
the front. For example: "dalek" would contain the substrings
{d, da, dal, dale, dalek}.
void filter :: search(string, set<track_id> &);
Parse the string into substrings following the "Parsing"
section (above). We want to find track_ids that match ALL
substrings, so take the intersection of all sets returned by
the filter_index for a given substring.
void filter :: print_cache_stats();
Print cache hit and size information.
void filter :: get_index();
Return the index storing all the filter data.
(Only available if -DCONFIG_TEST is set)
Groups: (lib/group.cpp)
Groups are going to be a new feature in Ocarina 6 and can compare
directly to Gmail-style labels. Ocarina 6 will create dynamic groups
that cannot be deleted by the user based on library status. Similar
to the library, groups should exist in their own namespace.
In Ocarina 6.0, groups are a wrapper around a specific index. Future
releases will store user-defined groups in a file on disk.
- Index:
Database<database :: IndexEntry> group_db
- Default groups:
All music
All tracks are added to this group
Library
Banned Songs
These groups are mutually exclusive. A track is either
in the Library or the Banned Songs group
- API
void group :: add(name, track_id)
group_idx.insert(name, track_id);
void group :: del(name, track_id)
grou_idx.delete(name, track_id)
void void group :: list(list<string> &);
return group_idx.keys();
void group :: get_tracks(name):
return group_idx[name]
Idle queue: (lib/idle.cpp)
The idle queue is used to schedule tasks to run at a later time. Idle
tasks must inherit from the IdleBase class so that multiple templated
IdleTask pointers can be placed on the same idle queue.
Creating an idle queue in idle.hpp will create a new queue for every
file that idle.h is included in. I want to have a single, shared
idle queue used by the entire application so to get around this the
IdleBase class is used and implemented in lib/idle.cpp.
- IdleBase:
class IdleBase {
private:
schedule();
public:
IdleBase();
~IdleBase();
virtual void run() = 0;
};
- IdleTask:
template <class T>
class IdleTask : IdleBase {
private:
void (*func)(T &);
T &data;
public:
IdleTask(void (*)(T &), T);
void run();
};
- Queue:
queue<IdleBase *> idle_queue;
float queued = 0.0
float serviced = 0.0
- API:
void IdleBase :: schedule();
Add the idle task to the idle queue. This should be called
by the IdleTask constructor.
queued++;
template <class T>
static inline void idle :: schedule(void (*)(T &), T);
Create a new IdleTask to run the function later.
bool idle :: run_task()
If there are tasks on the queue:
run the next task
scheduled++
If there are still tasks on the queue:
return true
else:
queued = 0
scheduled = 0
return false
float idle :: get_progress()
Return (serviced / queued) to the caller. If there are no
tasks, return 1.0 to indicate that the queue is finished (and
to avoid a divide-by-zero error).
Library: (lib/library.cpp)
The library manages databases containing track information added by the
user. Ocarina 6 splits the library into multiple database tables for
storing content. The library will exist in a library namespace to
to make functions and classes more unique.
When a library : Track is created, it should be added to the "Library"
group if it is NOT a member of the banned songs group.
- Databases:
enum DB_Type {
DB_ALBUM,
DB_ARTIST,
DB_GENRE,
DB_LIBRARY,
DB_TRACK,
};
- Album:
class library :: Album : public DatabaseEntry {
public:
string name;
unsigned int year;
unsigned int artist_id;
};
File << artist_id << year << name
- Artist:
class library :: Artist : public DatabaseEntry {
public:
string name;
};
File << name
- Genre:
class library :: Genre : public DatabaseEntry {
public:
string name;
};
File << name
- Library:
class library :: Library : public DatabaseEntry {
public:
string root_path;
bool enabled;
};
File << enabled << root_path
- Track:
class library :: Track : public DatabaseEntry {
public:
unsigned int library_id;
unsigned int artist_id;
unsigned int album_id;
unsigned int genre_id;
unsigned int track;
unsigned int last_year;
unsigned int last_month;
unsigned int last_day;
unsigned int play_count;
unsigned int length;
bool banned;
string title;
string length_str;
string filepath;
};
File << library_id << artist_id << album_id << genre_id << track << last_year
File << last_month << last_day << play_count << length << banned << endl
File << title << endl;
File << filepath << endl;
- Song:
struct Song {
library :: Album *album;
library :: Artist *artist;
library :: Genre *genre;
library :: Library *library;
library :: Track *track;
};
- Databases:
Database<library :: Album> album_db(album.db);
Database<library :: Artist> artist_db(artist.db);
Database<library :: Genre> genre_db(genre.db);
Database<library :: Library> library_db(library.db);
Database<library :: Track> track_db(track.db);
- Updating algorithm:
set<pair<lib_id, track_path>> known_tracks;
1) For each track currently in the library, check if the track exists
in the filesystem.
1a) If the track does exist, add to the known_tracks map.
1b) Else, mark track invalid.
2) For each file in the scan directory, check if (lib_id, track_path)
exists in the known_tracks map.
2a) If the file is in the map, do nothing.
2b) Else, add track to the library, to the groups "All Music" and
"Library", and then to the filter index.
3) Save all databases
The taglib library should be used for finding artist, album, etc. tags
for each track.
Use idle tasks for step 2 to break up tagging new files into chunks.
This way the user will still be able to use Ocarina and scanning can
happen while idle.
- Testing:
The script tests/library/gen_library.sh will create a sample library
in the /tmp/ directory for testing purposes. All the track files are
complete silence, but the script will fake up tags for each file.
- API
void library :: init();
Initialize databases and read files from disk. Fill out
groups and prepare filter as tracks are read.
bool library :: add_path(string dir);
If dir is not a directory:
return false
Add new row to the library_db table, begin an update only
on the new path.
return true
void library :: del_path(unsigned int lib_id);
Invalidate a library_db row and all tracks owned by that path
void library :: update_path(lib_id);
Update the given library_db row, if valid.
struct Song library :: lookup(track_id)
Fill out a Song structure for the provided track_id
#ifdef CONFIG_DEBUG
void library :: print_db(DB_Type);
Print the database corresponding to DB_Type
void library :: reset();
Clear all databases, returning the library to an empty state.
endif /* CONFIG_DEBUG */
Playlist: (lib/playlist.cpp)
Playlists are a list of songs that the user has configured to play. I
will create a pool of playlists that will be filled by user actions.
Playlists will be put on a "deck" that is used to give an order to the
next songs played. When deck :: next() is called, find the first
playlist with PL_ENABLED set and call that playlists next() function.
When a playlist is empty, remove it from the deck.
- Flags:
enum playlist_flags {
PL_ENABLED (1 << 0),
PL_RANDOM (1 << 1),
PL_DRAIN (1 << 2),
};
- Playlist:
class Playlist : public Database {
private:
database<track_id> tracks; /* Keyed on track id */
unsigned int cur;
unsigned int flags;
public:
Playlist();
void add(vector<track_id> &);
void del(vector<track_id> &);
void set_flag(playlist_flags);
const unsigned int get_flags();
unsigned int size()
File &operator<<(File &);
File &operator>>(File &);
void sort();
void next();
}
File << flags << tracks.size() << tracks[0] << tracks[1] << ... << tracks[N];
- Deck:
list<Playlist> deck;
unsigned int current_track;
File << current_track << deck.size() << endl;
File << deck[0] << endl;
File << deck[1] << endl;
- Deck:
list<Playlist> deck;
File << deck[0] << endl;
File << deck[1] << endl;
File << deck[N] << endl;
- API
deck :: init();
Read in the playlist file
deck :: new();
Adds a new playlist to the deck
deck :: rm(N)
Removes playlist N from the deck
Playlist *deck :: get(N)
Return playlist N from the deck
deck :: next()
Play the next song from the deck
- TODO <<<<<
What if each playlist has its own playlist_id for tracks? This would
allow for simpler removals, since I won't need to search for a track id.
I can easily create a function for mapping a list of playlist_ids to
track_ids...
- API
void playlist :: init();
Read in the playlist file
* Playlist *playlist :: add();
Add a new playlist to the deck, return the created playlist
to the caller.
* void playlist :: remove(Playlist *);
Remove the provided playlist from the deck. The pointer will
be unusable after calling this function.
void playlist :: move(Playlist *, unsigned int);
Move the playlist to the provided location in the deck.
* trackid_t playlist :: next()
Return the next trackid from the top playlist on the playlist
deck (id = deck[0].next()). If the top playlist is now empty,
remove it.
void playlist :: prev()
Keep a playlist :: Playlist recent(PL_ENABLED)
Whenever next() is called, add the returned track to the front
of this playlist, reset recent.cur to 0.
When prev() is called, return recent.next();
trackid_t playlist :: Playlist :: next()
If PL_RANDOM is set:
Randomly pick a value between 1 and size(). Increment
the cur pointer by this value, taking into account any
roll over.
Else:
cur += 1, if cur == size(): cur = 0;
if PL_DRAIN is set:
Remove the trackid pointed to by cur from the list and
return its value.
return list[cur]
void playlist :: Playlist :: set_flag(flag);
Set the user-requested flag
void playlist :: Playlist :: clear_flag(flag);
Clear the user-requested flag
const unsigned int playlist :: Playlist :: get_flags();
return flags
bool playlist :: Playlist :: play_row(unsigned int id);
Call this fuction to play a song from the playlist. id matches
up to the index in the playlist to play. Return true if the
selected row should be removed from the playlist, false
otherwise.
string playlist :: Playlist :: get_length()
Calculate the length of the playlist and return a string
in mm:ss format with the results.
unsigned int playlist :: Playlist :: size()
Return the number of tracks in your playlist
unsigned int playlist :: Playlist :: current_index()
Return the current index into the playlist
unsigned int playlist :: Playlist :: filter(string text)
Set the current filter text
unsigned int playlist :: Playlist :: is_visible(id)
Check if the playlist id is visible based on playlis text
* unsigned int playlist :: Playlist :: add_track(trackid_t)
Add a new track to the playlist
* unsigned int playlist :: Playlist :: rm_track(playlistid_t)
Remove a row from the playlist
Audio: (lib/audio.cpp)
This file will introduce an "audio" namespace containing all of the
functions interacting with gstreamer. This will create a wrapper
namespace that will be easier to work with than using raw gstreamer
functions.
The audio layer will also control the "pause after N tracks" feature
so songs can be loaded without neeting to pass in a "begin playback"
flag every time. The remaining tracks counter will only be decremented
when a song finishes through the end-of-stream message passed by the
gst pipeline.
- Internal:
Set up a message bus to look for end-of-stream and error messages so
the next song can be played. This function should call the play
function after loading a track and after checking the "pause after N"
count.
- API:
void audio :: init(argc, argv)
Initialize the gstreamer layer and reload the track that was
last loaded before shutdown. Please only pass --gst-* options
for argv.
void audio :: play()
Begin or resume playback.
void audio :: pause()
Pause playback.
void audio :: seek_to(int)
Seek to a position X seconds into the track
void audio :: stop()
pause()
seek_to(0)
void audio :: next()
Call the playlist :: next() function to get the next trackid,
and load that file into the gstreamer pipeline. Do not change
the state of the pipeline (if nothing is playing yet, don't
call play()).
void audio :: previous()
Call the playlist :: previous() function to iterate backwards
through the recently played playlist. Load the returned trackid
without changing the pipeline state.
trackid audio :: current_trackid()
Return the trackid of the currently playing song.
unsigned int audio :: position()
Return the number of seconds that the song has played.
unsigned int audio :: duration()
Return the duration of the current song in seconds.
void audio :: pause_after(unsigned int)
Pause after N tracks, pass a negative number to disable.
unsigned int audio :: pause_count()
Return the number of tracks that will be played before
playback pauses.
Gui: (ocarina/*)
The GUI will be written in C++ using gtkmm3 for (hopefully) cleaner code.
- Design requirements:
- gtkmm3
- Front-end for the library to add, remove and modify paths
- This can be hidden since it's not a common task
- Double-click to play songs
- Front-end for groups to add and remove tracks
- This is a common task and should not be hidden
- Don't save / restore window size
- Some window managers already do this automatically
- Set reasonable defaults
Future work:
I want to set reasonable expectations for Ocarina 6 so that I don't
have to spend a large amount of time coding before releasing something
to the wild. This section will be a list of features that I want, but
should be deferred to a future release so basic support can be coded.
Hint: If feature B depends on A, implement A in 6.x and B in 6.x+1
- New default groups: (6.1)
Unplayed tracks
- Categories: (6.1)
Use these to make "groups of groups" for better organization.
Different categories can include Album, Artist and Genere
dynamic groups in addition to user created groups (see below)
The Artist, Album and Genre "tables" can be used to populate
these categories.
- User created song groups: (6.2)
Basic add and remove features can be implemented using the
Library and Banned Songs groups. This will give me a chance
to test saving groups on a small scale before letting users
create anything they want.
- Save a user's playlist as a group: (6.2)
- Library defragment: (6.1)
Ocarina 6.0 will leave holes in the library when tracks are
deleted, potentially leading to fragmentation and larger-than-
needed file sizes. A "defragment" utility can be created to
clean up unused slots.
To help with fixing groups, a mapping of (old values) ->
(new values) should be kept.
- Fix track durations: (6.1)
Some tracks in my library are tagged with the wrong duration,
so fix them as they are played.
- Track tag editor: (6.2)
Make a pop-up window for editing the tags of a track. Be sure
to update the library information and the on-disk file.
- Album art: (6.1)
- Playlist custom sorting: (6.1)
Click column headers to choos sort order
Keep a list of fields that the user has selected and place new
fields in the front of this list. Use a recursive stable sort
to do the sorting.
- Better design file format: (6.1)
Todo list in each document instead of all at once in the footer.
Leave the Todo list out of the official "design document" and
keep it in each individual section instead.
XML?
Code formatting?
- Copy a song group to a different directory: (6.x)
This can be useful for keeping an external device (like an
Android phone) updated with the music you want to listen to.
Complications: I have an mp3 mirror of all my music, and I
want the mp3s to be synced. Perhaps track mirrors in Ocarina?
- Mirror directory: (6.x)
I rip music to .flac, but keep an mp3 mirror directory to sync
to other computers and phones. An Ocarina tool to manage a
COMPLETE library mirror might be a good idea so I no longer
need to manage it externally. This can still be done with a
script, a cron job, and maybe a "mirror this track" option in
the library? Perhaps create a mirror group?
- Extra testing ideas: (6.1)
- Run tests through valgrind to help find memory leaks
- Combine earlier tests into a single file
- Exceptions: (6.1)
- Don't return error codes, throw exceptions like C++ is
designed to do.