Search in Add Card Dialog

What it's about This add-on aims to add some search functionality to the Add dialog. I am writing this for personal use, but maybe someone out there might profit of it too. I wanted a functionality like this because I don't like the default add card dialog, which is like a black box for me where I just insert my new information. I prefer to have some context, e.g. what other notes I already have that might be related to what I am currently writing. Another use case would be browsing my existing notes, determining the notes that need improvement (bad retention, long answer times) or even the topics (represented by tags) where I am struggling, and while I do that, I want to be able to add new notes or to edit the existing ones. Bonus point is that I can learn html/js/css by coding this :-) What the add-on does is to build a search index out of your decks, which can then be queried. When you type, it tries to find the note whose text is the most similar to what you have typed or selected. Because naturally, many common words are present in a lot of notes, there is a list of words that should be ignored (so called stop words). Examples in english would be "the", "a", "I", "for", etc. I cannot know what languages you write your notes in, so you have to build it by yourself. The default configuration contains an english stop word list for starters. The stop word list is defined in the config file, and you should fit it to your language(s) before using this add-on. Your searching will take longer and be less precise if you don't use stop words. All UI concerning the search is displayed on the right side of the dialog (it is only displayed above a certain window width and height, so don't be confused if it may not be there when you start). The index is built everytime you start Anki, and by default uses your whole collection. At the moment, notes you delete are not deleted from the search index until you restart. Same goes for notes you edit during review or in the browser. Notes you edit through the edit button in the Add Card dialog are updated immediately. Features atm include: - Selecting text inside a field or inside a displayed note allows to search for it - Automated searching if you type, or if you input tags - Edit another note in the add window - Timeline to see how productive you have been in the current year (CTRL + hovering will show you the cards added on that day) - Predefined searches - Info dialog for displayed notes - Browser search mask that works exactly like the search you know from the browser window - Defining synonyms/related terms to include when searching - Defining stopwords to ignore when searching - Defining fields to ignore when searching or to hide in the results - Browsing/searching for tags - Browsing related tags and their notes by hovering over a tag - Toggling between normal add dialog view and the add-on (by default CTRL-F) Toggled: (press CTRL-F to switch to the search pane) Toggled the other way + grid + sidebar hidden: Editing: "Timeline" (CTRL + hover to see the info): Tag info (a little wonky atm): Night mode styling: Explanation of the options: deck select - only cards from the selected decks are returned as results search on select - if activated, selecting some text (with the mouse!) in the input fields on the left will trigger a search search on typing - if activated, typing in the input fields on the left will trigger a search. you may want to disable this if you feel that search is slowing down the typing or distracting you. search on tag entry - if activated, typing in the tag input will bring up all notes (atm from all decks, not restricted by your current deck selection) that have at least one of the entered tag. Works with parts of hierarchical tags to. highlight results - if activated, matched words are highlighted yellow (does not work in 100% of cases currently, but mostly) freeze - freeze the current results random - will display a number of random notes from your currently selected decks, amount is determined by what you have set numberOfResults to. The search results are updated when a field on the left is changed, if a query is entered through the bottom search mask, or if text is selected on the left. Results are notes, not cards, where fields are separated by a | character. Clicking on a note will show some (very basic) evaluation on how you performed on card(s) of the note. Clicking on a tag of the note will add the tag to the currently edited note. Results can be pinned / removed (this will only remove the displayed result, NOT delete anything!). Support for synonyms / query expansion (24-03-19) If you enter "What is Anki?" on the left side of the dialog, your search results will only contain notes with "anki" in them. But you might want to retrieve related notes too, that do not contain the term "anki", but maybe something like "spaced repetition". Another use case would be word inflection, because currently, stemming is not activated in the index (that would be language dependent), which means searching for "space" and "spacing" will get different results. If you click on the small s icon on the upper right, you will see a dialog that allows for the input of synonym sets. These are used when searching, but only for the search during typing or selection, not for the browser search. Terms are separated by ",". If you click inside a set, you can edit it and confirm by pressing enter. If you use synonyms, searching works like follows: - Suppose you typed "What is Anki?" - Your stopword list contains "what" and "is", so these are removed - One of your synonym sets contains "anki", so the query gets expanded by all the terms in the set - End query: "anki srs spaced repetition" You should maybe use this conservatively, since you might see too many results for simple queries otherwise. Browse Tags (12-04-19) You should now be able to browse the tags in your collection from the deck select. Clicking on a tag will find all notes (unrestricted by your current deck select, but by numberOfResults) that contain the tag. If your note has the tag top::middle::bottom and you click on either top, middle or bottom in the list, the note will show up. Experimental: Draggable notes (16-04) Added a button to make a note draggable. This is experimental and I am not sure if this is at all useful. You can drag the notes to the left if you need to keep their content visible and you don't want to pin them. Dragging is a bit sluggish for me on Windows, but it kinda works. (Resizing the window or dragging out of the window boundaries produces some weird behaviour though). A dragged note is like a pinned note, so if you have it displayed, and search for some of its contents, it won't appear in the search results. Config options: decks: here you can specify which decks should be used to build the index. If none are given, all are used. This can be useful, e.g. I have a giant premade deck of chinese vocab (6k notes I think), which I don't want to have in the index (because basically every english word is contained there). Names must exactly match the respective deck names. stopwords: stop words are terms that are not very meaningful concerning the relevancy of a document, simply because they are too common, e.g. 'in', 'a', 'the',... These words can/should be ignored when searching. If your language is different from english like mine or you are writing notes in multiple languages, I absolutely recommend building your own list here. Especially if you use the default SQLite FTS index, which seems to be noticeably slower without a good stopword list. numberOfResults: how many search results should be returned (1-500 are valid). This value is ignored by the predefined searches as you can set your own limit there, but for search on typing/selection/newest notes for a tag/notes added in the timeline, this value determines the max. number of displayed notes. A higher value might result in a more 'laggy' feel when using search on typing, as the typing typically lags a bit while the results are rendered, and more results means higher render time. useFTS: There are two possibilities to search: By default, sqlite fts (short for full text search) is used, but there is also the possiblity of using the included library Whoosh for that. There are different versions of SQLite FTS, the newest being FTS5. If you set this option to true, on startup the version available will be checked. If 5 cannot be used, 3/4 are used instead. I noticed that the index builds much faster on the fts versions than on Whoosh, however, the relevance ranking on fts4/3 is currently buggy, so your mileage may vary if you don't have fts5. I am on windows 10, and fts5 is included in my sqlite install, and I actually prefer this mode over Whoosh because of the faster startup. I suggest to at least try out fts, escpecially if your collection is huge. addToResultAreaHeight: If you see a vertical scrollbar at the right side of your window, most likely the result area (where the found notes are displayed) is too high. I suck at css and have not succeeded to make this always fit your exact window size. So for now, I included this lazy solution which lets you modify the height of the result area. Accepted values are -500 to 500 (without quotes). -100 means subtract 100 pixels from the height. delayWhileTyping: This is the amount of milliseconds, which are waited for until a search is triggered. The default value of 1000 means that if the user has stopped typing for 1 sec, a search is triggered. Minimal value is 500, I suggest not to set this too low because the typing might feel laggy otherwise. useInEdit: If set to true, the search interface can be used from the edit dialog (the one which can be opened during review) too. Should behave exactly like in the Add Card dialog. logging: If enabled, will print some information to log.txt in the addon folder. I used this for debugging. You should leave this disabled unless you have some problem and want to send me some additional information. fieldsToExclude: Accepts a map of note type name : [ field name ]. "fieldsToExclude" : { "myNoteTypeName" : [ "FieldToExclude1", "FieldToExclude2" ], ... } This will still show the fields in the output, they will be only ignored on searching. Because the highlighting is independent from the search process, the words in a field you excluded from search might be highlighted, but that does not mean that the field was actually searched. Styling: I added options to set the colors, backgrounds and font-sizes for some elements, so if you run night mode, you can adjust these. Colors accept everything you could write in CSS, e.g. "rgb(0,0,0)", "#ffffff", and "blue". I did not exhaustively check if all UI elements can be styled that way as I do not use night mode myself, so message me if I forgot something. In case I forgot some option here also check the changelog below. Some notes: IMPORTANT: I have no possibility to test on Mac, so I have no clue how things look there. So if something is off, please tell me, otherwise it will probably never be fixed. 1. Whoosh index typically takes more time to build (for me 30s to 50s, might be even longer for people with larger collections). I recommend using the SQLite FTS index (performance on this one varies depending on whether you can use version 3, 4 or 5). 2. Cards edited through the browser or deleted in the current session are not in the index/removed from the index until you restart the program. 3. I could only test how my cards look in the results, I have little media and mostly text, although the few images I had were displayed fine. There might be problems with very large images. 4. Mathjax is not rendered at the moment. 5. If you run Anki with the nightmode add-on, you can play with the config options for colors/backgrounds, but keep in mind notes are not inverted, so images with white background will still be white. 6. I don't think it will work well/at all with languages like chinese, that don't separate words on whitespace. 7. I noticed my search time rapidly goes up if the text contains lots of chinese characters. 8. No warranties about compatibility with other add-ons. I personally don't use that many, and it would be exhausting to test all possible combinations, so you have to find out by yourself. EDIT: I just realized I can't respond to bug reports here, so either post an issue here: https://github.com/fonol/anki-search-inside-add-card or, in case you have a reddit account, please send me a message (or comment in this thread: https://www.reddit.com/r/Anki/comments/axpvjf/search_collection_inside_add_cards_dialog/). Username is Brunzus. I will try to look after all bugs as soon as I have time. FAQ I want to change the looks of the addon. -> There is a config section called styling, where most colors can be set. If you want to switch between normal and night mode, there is a predefined styling config if you click on the Settings button, which you can simply copy paste into your config.json. When I open the add dialog for the first time, index creation takes too long. -> If you are not already using it, see notes point 1 above, try out the fts index. Otherwise, there is not much I can do about that, performance depends on your machine and the size of your collection. You can always exclude decks to reduce the index size. A note in the results seems to be displayed incorrectly (text flows out of the card). -> Some notes might contain html that breaks the container into which they are rendered. I try to catch these cases, but if you stumble upon such a note, please open an issue in the github repo with the note text. Sometimes a search is not triggered when selecting -> The search is triggered when the mouse up happens inside (!) the note (or field input). So if you mark some text, and leave the note area before releasing the mouse, nothing happens. I get ugly scrollbars in the right side of the add card dialog. -> I try my best to make the column on the right side always fill exactly 100% of the height, but I only succeeded on windows atm. So I included the option addToResultAreaHeight in the config file. See in the config section for an explanation. I have a note with lots of text, and typing seems to lag at some point -> Typing typically lags when the results are rendered. My advice would be to either increase the delayWhileTyping config value (I am a slow typer and have mine set to 1500ms atm), so searches aren't triggered too soon while typing, or to uncheck search while typing. I set useFTS to false, and get an import error when starting Anki -> If you are on Mac OS, your Anki is missing a library for Whoosh (the other type of search index that may be used). Atm there is no solution for this, so you have to stick with sqlite fts. What is this Performance stat? -> I wanted to have a stat that includes not only the pass rate of a note's cards, but also how long I took on average to answer these cards. So I simply came up with a crude formula that calculates a score (1-100) out of the pass rate and avg. time taken. I am open to suggestions about better stats to display there instead, so if you have any ideas, let me know. How are the keywords determined? -> The keywords are not really determined in a sophisticated way, it's just the words that appear most often in the top n results (n = 20 atm) , minus any stopwords. CHANGELOG: 13-04-19: + added config options for colors and font-sizes 14-04-19: + added search button for last added notes 15-04-19: + added created date and search for notes created on the same day (by clicking on ranking number) 16-04-19: + added experimental feature: make note from results draggable (see above) + added config option for logging (activate if something doesn't work and you want to send me more info) - fix "Created x minutes ago" displaying floating point numbers - fix result rendering throwing an error for some users 17-04-19: + small UI changes + info dialog now displays exact FTS version (distinguish between 3, 4, 5 and none) - possible fix: if FTS4 is not detected, use FTS3, this might fix indexing not working for some linux users 18-04-19: + added some small status that is displayed while searching + searches that contain > 2000 chars are aborted now - fix disabled overflow when note contains lots of fields - did some changes to highlighting, it is more accurate now, but kinda slow on text with accented characters 19-04-19: + improvements for Whoosh: Search is now performed in background like fts, highlighting works better + added small info at startup + if you change your deck selection, last search should be repeated with the new selection + config useInEdit (experimental): if enabled, search is available in the edit window that can be opened during review 21-04-19: + selection and typing queries are now profiled, you can view the execution time by clicking on the ⓘ icon next to the time + improved speed of fts3 and fts4 queries + marked words next to each other should now be marked together - fix pinned items not being cleared when dialog is closed and opened again 27-04-19: + notes modified through the edit button are now updated in the search index (should work for Whoosh and SQLite FTS) - fix deck selector toggling itself after adding a card - fix entering tags in edit dialog triggering a search - fix backspace not triggering search on typing 30-04-19: + did some changes to the displayed info on notes (now opens in modal, which I like more because of more available space) + info on a note now shows a simple revision plot + added grid view (two columns, just checked and it doesn't seem to work on Ubuntu at first for some reason, but after toggling it on and off like 2-3 times it displayed correctly, will update when I found a fix) - fixed an error when loading the index and closing the dialog during loading 02-05-19: - fix error when closing add dialog while having an open edit window - fix note in the results being duplicated when result count is an odd number - fix note results rendering slowing down after doing a lot of searches 08-05-19: + add some more predefined searches EDIT: just tested it on my normal Anki installation, and the dropdown is totally sluggish. I don't know why, on my debugging environment, it ran perfectly smooth. For now, I'm gonna leave it like that until I found the reason. - fix collapse icon being reverted when add dialog is closed and top section is hidden - fix closing edit dialog with enter not updating displayed note 12-05-19: - fix notes in grid view not reflowing correctly when a note is removed from the results - tiny UI fixes 27-05-19: + added option to render the results immediately, instead of fading them in: renderImmediately + added option to make clicks on a tag search for notes with the selected tag: tagClickShouldSearch + decreased default font size of deck select - fix error when unselecting all decks and "last added search" is repeated 30-05-19: + added config option to control the window split (thanks to yzhang1337 for the idea): leftSideWidthInPercent controls how big the left column is, e.g. a value of 40 means a split of 40/60 + hide tags and keywords if window is small, to leave more room for the results + added option to show the retention for each result (small colored label under the result number): showRetentionScores + hovering over a tag in the results shows some info and related tags (if you hover longer than 300ms). This is still a WIP, it bugs sometimes and shows on the wrong side, but basically it works + added imageMaxHeight, setting this to e.g. 500 means all images in the results take at max 500px in height. 01-06-19: + reworked the bottom bar a bit, should be a bit prettier now + added some sorting/filtering to the bottom bar + added option to switch between search and normal pane when window is small - fix floating note not being deleted from list - fix grid not reflowing when making a note floating 02-06-19: + added support for chinese and japanese characters + improved positioning of the tag info a bit 03-06-19: + added a modal to set some stylings, mostly to point out that there is AddToResultAreaHeight, because the correct sizing seems to break easily on different screens 04-06-19: + added interval and answer times graph to the info modal - fix stats being mixed together for cards with same names in info modal 05-06-19: + added config option to hide the sidebar + try to hide images generated through the image occlusion add-on - fix loader sometimes shown when reopening add dialog 30-06-19: + added some predefined searches: last lapses, last reviewed, and time taken + right clicking on a note opens a context menu, which has not much content for now, maybe some actions will be added in the future + right clicking on an image allows to copy it to a field - if a search returns no results, the old results are not emptied anymore 01-07-19: + added toggle shortcut: CTRL-F (changeable through the config) - fix images sometimes overflowing their note in grid view 02-07-19: - info modal should now resize better when the window is small - don't show the tag info on hover if the window is small - grid/freeze/random buttons now hide their texts in small windows 04-07-19: + added a small timeline to the bottom, which displays each day of the year in a color depending on how many cards where added that day, still a bit rough, but it works. You can hover to see the cards on each given day. + improved the layout a bit, there might be less issues with the height of the right side being not correctly set + if you have the window in a large size, and you hide the right side with CTRL-F, the state should be remembered on closing the dialog 05-07-19: + added config option to hide the timeline (available also through the styling modal) + added config option to hide the tag hover info (available also through the styling modal) + some minor ui updates 12-07-19: - fix right layout being destroyed after note is added and top menu is hidden - fix last day in timeline being colored wrong 17-07-19: + added a config option to exlude fields: fieldsToExclude, see above for a short explanation 21-07-19: - fix timeline sometimes marking the previous day as the current day - fix error thrown when search on tag entry is enabled and the entered tag contains ' single quotes - fix some accented characters not being searched or being ignored on highlighting - include fewer related tags in the tag hover info if the related tags exceed a certain total length 29-07-19: + some ui changes in the info dialog + added a small graph in the info dialog that displays the reviews/intervals for each card + lapses are marked in the info dialog - renamed retention to pass rate - hopefully the last fix for the timeline 03-08-19: + if showTagInfoOnHover is enabled, and you hover over a tag, assuming the window is large enough, the tag info box also contains the newest notes for the tag. If you hover over related tags again, you can nest these windows. (I included a screenshot above). Clicking on one of the windows will close all others. + you can now search for a synset in the synset dialog - hopefully the absolutely last fix for the timeline 05-08-19: + Hovering tags on the left side of the tag info box now opens another box too, like in the 'related' section + 'mytag (+1)' is now shown as 'mytag::subtag' + Added an option tagHoverDelayInMiliSec, which determines how long to hover over a tag until the tag info is rendered. Editable through the styling modal. - fix related tags in some cases not being correctly determined - fix '${}' in the note text causing an error in the tag info - fix some of the filtering options not working after a filter has been applied that would have removed all results - fix styling modal throwing an error if input field is empty 07-08-19: + added a config option to disable printing of excluded fields: showExcludedFields + try to show fields with just a single image in their own line - changed default value of imageMaxHeight to 100 px - fix image occlusion images being rendered after note has been edited 08-08-19: + reworked the styling part of the config, all colors/font-sizes are now under a section called "styling", so you might need to reset your config.json, but absolutely save your stopwords/synsets before, as they will be reset too! + I included a sample styling section for night mode compatibility in the Info dialog - fix grid view icon not being colored blue after reopening add dialog 10-08-19: + added a Rebuild Index button under Info + try to not always rebuild the index, this works for most of the cases, except for edited notes outside the add-on pane and for notes that changed decks. In those cases, you can use Rebuild Index. If you want to disable this feature, you can set alwaysRebuildIndexIfSmallerThan to some high value. + added alwaysRebuildIndexIfSmallerThan config option, should be self-explanatory 14-08-19: + removed showExcludedFields config option, excluded or hidden fields can now be set independently under Settings -> Set Fields + added pagination, this means a better performance (only have to highlight the current page, have to render less html overall), this also means you should set numberOfResults to a higher value, I set the default to 1000. + because pagination and timeline sit close to each other, I disabled the timeline hover by mouse, you can either press CTRL and hover to get the same behaviour, or click on a single day. - fix some logic errors in the display of hierarchical tags 15-08-19: - fix showExcludedFields still being output in the index dialog 16-08-19: - fix error after pasting image into empty field - fix tag modal not appearing on tags with "..." 22-08-19: - fix some display issue with buttons on deck select - pinned notes should now work better with the grid view