-
Feanil Patel authored
* OeX_ES-33: Use django-es-dsl into search model methods. Set ordering by id for Program model. * OeX_ES-33: Uses django-es-dsl for course runs views. * OeX_ES-33: Uses django-es-dsl for catalog queries views. * OeX_ES-33: Removes haystack and uses django-es-dsl for catalogs models. * OeX_ES-33: For clean query utils handle the case when query string contains a whitespase * OeX_ES-33: Adds customs text analyzers. Particularly 'html_strip', 'synonym_text'. * OeX_ES-33: Adds ES course document. * OeX_ES-33: Adds ES course run document. * OeX_ES-33: Adds ES person document. * OeX_ES-33: Adds ES program document. * OeX_ES-33: Common indexes logic. * OeX_ES-33: determs Elasticsearch documents package * OeX_ES-33: Updates course runs view test module * OeX_ES-33: Updates programs view test module * OeX_ES-33: Updates search view test module * OeX_ES-33: Updates tests utils module from core app to cut off haystack. * OeX_ES-33: Adds a fix to SearchQuerySetWrapper to provide es document by index. * OeX_ES-33: Fix for CourseRun model to be able search via '(*)'. * OeX_ES-33: Updates test_models test module of course_metadata app. * OeX_ES-33: Updates test_signals test module of course_metadata app. * OeX_ES-33: For es PersonDocument rebuilt position field generation. * OeX_ES-33: For es ProgramDocument fix for prepare sit types. And subject_uuids, authoring_organization_uuids, staff_uuids has to be keyword instead of text. * OeX_ES-33: Updates test_serializers test module of api app. * OeX_ES-33: Creates generic serializers mixins to handle es response. ModelObjectDocumentSerializerMixin and DocumentDSLSerializerMixin. * OeX_ES-33: Describes es serializers package interface. * OeX_ES-33: Adds serializer for course document for elasticsearch response. * OeX_ES-33: Adds serializer for course run document for elasticsearch response. * OeX_ES-33: Adds serializer for person document for elasticsearch response. * OeX_ES-33: Adds serializer for program document for elasticsearch response. * OeX_ES-33: Adds constants for search indexes. * OeX_ES-33: Adds custom elasticsearch real time signal processor. * OeX_ES-33: Changes template text file name to be able use old es text templates. * OeX_ES-33: Mixin class for supporting faceting for into elastissearch on an API View. * OeX_ES-33: Fix for elasticsearch boost config. Cause negative is not supported anymore. * OeX_ES-33: In order to es7 synonyms have to be list of string with comma separated values. * OeX_ES-33: Cut off and rebuilt es for main conftest file to be able get es connection and create/deletes es indexes. * OeX_ES-33: Updates ElasticsearchTestMixin. * OeX_ES-33: Renames elasticsearch dsl default connection fixture. * OeX_ES-33: Updates test view mixins. * OeX_ES-33: Moves search engine serializers to search_indexes module. Cut off haystack from api/serializers. * OeX_ES-33: Fixes test serializers module. * OeX_ES-33: Refactores search indexes module serializers. * OeX_ES-33: Updates test search module. * OeX_ES-33: Updates all es search viewsets. * OeX_ES-33: Updates test create index. Deletes test alias exists cause django es dsl has own manage.py commands to manage indexes. * OeX_ES-33: adds aliases for es indexes. * OeX_ES-33: Puts changes to function_score which are relevant to es7 requirements. * OeX_ES-33: Updates test boosting module of edx haystack extensions app. * OeX_ES-33: Extends Document class of boosting search * OeX_ES-33: For ProgramDocument changes fiels type for attribute. * OeX_ES-33: Deletes haystacks serializers from aggregate facet search serializers. * OeX_ES-33: Updates edx catalog extensions serializers. * OeX_ES-33: Updates faceted search fields for attribute * OeX_ES-33: Updates edx catalog extensions test views tests. * OeX_ES-33: Adds custom FacetedFieldSearchFilterBackend to Facet view mixin. * OeX_ES-33: Modifies distinct count module. Due to architectural aspects of elasticsearch dsl removed old haystack QueryBackend and moved the logic to the customized Search class. * OeX_ES-33: Uses constant from search indexes package * OeX_ES-33: Adds a module of django elasticsearch dsl drf specific serializers. * OeX_ES-33: Refactoring due removing custom response class to separete module. * OeX_ES-33: Adds module with extended elasticsearch-dsl responses. * OeX_ES-33: Adds module with extended elasticsearch-dsl Search classes. * OeX_ES-33: Adds module with extended django-elasticsearch-dsl-drf viewsets. Also implements BaseElasticsearchDocumentViewSet. * OeX_ES-33: Adds module with extended django-elasticsearch-dsl-drf backends. * OeX_ES-33: Deletes a module with hastack implementation of es indexes. It's really so sad. but time is up for them. * OeX_ES-33: Renames edx_haystack_extensions application name on a new one edx_elasticsearch_dsl_extensions. * OeX_ES-33: Handles the case where query value already coming in quoted. * OeX_ES-33: Adds edge_ngram_completion custom analyzer. * OeX_ES-33: Specifies addition settings for all es indexes. * OeX_ES-33: Updates course-discovery settings. * OeX_ES-33: Deletes not related now drf filters and their all tests. * OeX_ES-33: Elasticsearch dsl add xdist suffix to index name works now. * OeX_ES-33: Moves common filter backends(MultiMatchSearchFilterBackend, DefaultOrderingFilterBackend) to BaseElasticsearchDocumentViewSet. * OeX_ES-33: Fix for multi document serializer mixin to be able work in mulri processes mode. * OeX_ES-33: Uses ELASTICSEARCH_DSL_SIGNAL_PROCESSOR instead of HAYSTACK_SIGNAL_PROCESSOR for UpdateIndexTests. * OeX_ES-33: Deletes from django INSTALLED_APPS. * OeX_ES-33: Refactoring. For CourseRunTests test suit use elasticsearch_dsl_default_connection fixture instead of manage.py commands to create es indexes. * OeX_ES-33: Updates Tracis CI docker-compose yaml in es7 perspective. * OeX_ES-33: Fix for test course runs module. * Revert "OeX_ES-33: Fix for test course runs module." This reverts commit a06bb4ccded65756ac08665ac77c147e5ef1a521. * OeX_ES-33: Adds get_latest_by field to Program model explicitly. * OeX_ES-33: To be 100% sure, deletes all es indexes after a test session. * OeX_ES-33: Just in case deletes all index before test to have clean conditions. * OeX_ES-33: Updates ELASTICSEARCH_INDEX_NAMES in multiprocesses test execution mode too. The names have to be updated. * OeX_ES-33: Adds additional request value due travis add one request for own purposes. * OeX_ES-33: Style check. Reordes imports. * OeX_ES-33: Modefies paid_seat_enrollment_end boost to still to be able to penalize course runs without enrollable, paid seats. * OeX_ES-33: Adds ELASTICSEARCH_DSL_INDEX_RETENTION_LIMIT to base settings. * OeX_ES-33: Updates command to be able use it when there are four indexes and no haystack lib. * OeX_ES-33: Updates tests for the command . * OeX_ES-33: Deletes senseless alias from all es indexes. * OeX_ES-33: Update index utils. * OeX_ES-33: Rebuilds clear_es)indexes fixtute to be able delete all indexes after a test session by alias prefix. * OeX_ES-33: Adds threshold to test signals module cause Travis sometimes adds own db requests. * OeX_ES-33: Rebuilds command . Now the command bases on django-es-dsl. * OeX_ES-33: Deletes useless test mixins. * OeX_ES-33: Updates tests for command. * OeX_ES-33: Impemented feature in context index creation and updation. Add a new commands to search_index genaral django-elasticsearch-dsl managments. * OeX_ES-33: Removes action handler from overwritten search_index command. * OeX_ES-33: Style check fixes. * OeX_ES-33: Upgrades course-discovery requirements. Deletes django-hastack and drf-hastack from requirements and service at all. Goodbye, you would be great tools and fighters. Thanks for all! We will miss you. * OeX_ES-33: Adds additional style check fixes. * OeX_ES-33: Updates test boosting module to be able put back the case when course runs with expired paid seats are penalized. * OeX_ES-33: Pins pacing type for CourseRun model factory during test_typeahead_org_course_runs_come_up_first. And add message about an index creating for search_index --create command. * OeX_ES-33: Adds method to Elasticsearch global utils. * OeX_ES-33: Removes logic about form overwtitten command. * OeX_ES-33: Puts back command. Now the sanity check feature works as before. * OeX_ES-33: Updates update index tests cause new(old) command was developed. * OeX_ES-33: Fix import orders. * OeX_ES-33: Updates README. Adds new discription about base django-elasticsearch-dsl commands. Adds info about command. * OeX_ES-33: Refactoring for es documents. Moves all common fields to Base document. * OeX_ES-33: Code polishing. * OeX_ES-33: Fix for CourseRunViewSetTests test suit. * OeX_ES-33: Adds migrations. Mostly due to changing Program's default ordering, but also looks like some parler translation model field changes happened. * OeX_ES-33: After an index creation assigns to internal index name alias name. * OeX_ES-33: Deletes filter tokenizer from synonym text analyzer. * OeX_ES-33: Index matching by alias not by index name as was before. * OeX_ES-33: Extends and to be able delete indexes by index name cause delete by alias is impossible in es. * OeX_ES-33: Improves, fixes and simplifies command. * OeX_ES-33: Updates tests for and manage.py commands. * OeX_ES-33: Isort fix. * OeX_ES-33: Pylint fixes. * OeX_ES-33: Code polishing after a code review session. * OeX_ES-33: Fix for aggregation. * OeX_ES-33: Merged migrations * OeX_ES-33: Fix tests after rebasing on upstream master * OeX_ES-33: Deletes threshold for test_draft_does_not_collide_with_draft test. * OeX_ES-33: Pull data from ELASTICSEARCH_CLUSTER_URL instead of ELASTICSEARCH_URL to make cutover easier. * Revert "OeX_ES-33: Deletes threshold for test_draft_does_not_collide_with_draft test." This reverts commit a24af04fdd4e800573d7c4e58b16502c9184812e. * OeX_ES-33: Uses FuzzyInt instead of threshold value for 'test_collision_does_not_include_drafts' test. * OeX_ES-57: Reworks 'clean_query' function. Deletes lowercase transformation there since now it does elasticsearch itself. * OeX_ES-57: Adds new tests to cover search by key for all-list view. * OeX_ES-57: Adds search by 'key' and 'ord' fields. * OeX_ES-57: Enables 'analyze_wildcard' option for query_string type of search. * OeX_ES-57: Code polishing. * OeX_ES-57: Fixes problem when query value contains ':' character. To fix it override split lookup name for base search filter backend. New character is ':='. * OeX_ES-57: Adds for each request which is handled by django-es-dsl-drf view 'collapse' feature. Reason: attemp to fight 'duplicate issue' during es search. * OeX_ES-57: Refactoring. Moves 'prepare_authoring_organization_uuids' method to base document. * OeX_ES-57: Deletes key field from es templates. This is no longer necessary. Takes only redundancy. * OeX_ES-57: Extends multi match search firlds. Now we are able to search by key separately. * OeX_ES-57: Refactors es search views to add 'authoring_organization_uuids' fiels to filter fields. Also adds LimitedAggregateSearchView view filter backends. * OeX_ES-57: Extends LimitedAggregateSearchViewSetTests test suit to test the search results include the authoring_organization_uuids for each document. * OeX_ES-57: Adds FilteringFilterBackend for LimitedAggregateSearchView view. * OeX_ES-57: Isort fix. * OeX_ES-57: Pylint fixes. * OeX_ES-57: Adds additional lost fields to Course document. * OeX_ES-57: Changes multi match strategy. Now let's try 'phrase' match. * OeX_ES-57: Moves the logic about datetime serializetion to separete Mixin and use it. * OeX_ES-57: Add a new test to catalog suit to check courses with different catalog queries but the same meaning * OeX_ES-57: Add a new test to catalog suit to check courses with time range query. * OeX_ES-57: Add a new test to catalog suit to check courses with subjects and negative query. * OeX_ES-57: Uses for tests based on key the key which was used in a live debug session. * OeX_ES-57: Fix for Multi Document Serializer to be able use its own fields for an each specific serializer. * OeX_ES-57: Adds test to be sure that that draft objects are not indexed when run command to populate index. * OeX_ES-57: Adds test to be sure that that draft objects are not indexed when es indexes are being updated via django's signals. * OeX_ES-57: ES documents has not use django default model manager. Has to use particular one determined of the application. * OeX_ES-57: Fix. Multi document serializer has to define patch necessity when 'field' attr exist and not none. * OeX_ES-57: Do not use separete fields for AggregateSearchSerializer. * OeX_ES-57: Adds index update handler via django's signal when need to prevent indexing in the case when object is Draft. * OeX_ES-57: Fix import orders. * OeX_ES-57: Semgrep django style fixes. * OeX_ES-57: Raises IndexForbiddenException if es index is impossible. * OeX_ES-57: Pylint fixes. * OeX_ES-57: Serialize datetime fix for test serializers. * OeX_ES-57: Cause Draft object indexing is disabled now I have to change queries numbers for test_draft_does_not_collide_with_draft. * OeX_ES-57: Refuses collapsing as additional parametr for each es query. * OeX_ES-57: Puts ELASTICSEARCH_DSL_LOAD_PER_QUERY to course discovery settings. * OeX_ES-57: Adds default pagination to Extended es dsl Search class. * OeX_ES-57: Refactors common es document module. * OeX_ES-57: Adds test suit TestSearchPagination for es documents. * OeX_ES-57: Isort quality fixes. * OeX_ES-57: Bug fix. ES count request does not support paginated payload. So do not insert from and size to the request. * Change catalog test casing. - Org names can be a different case than the search query says. - The same is probably also true for the subjects so adding that to the test as well. * OeX_ES-57: Make for course based documents org field text and keyword types. * OeX_ES-57: Changes facet field for 'org' on 'org.raw' * OeX_ES-57: Deletes all courses to make clear environment for test_courses_with_subjects_and_negative_query test. * OeX_ES-57: Defines a module for specific custom exceptions for Elasticsearch. * OeX_ES-57: Fix for Elasticsearch search phase execution exception. * OeX_ES-57: Adds a django mixin to catch invalid Elasticsearch query string exception. * OeX_ES-57: Adds tests for the fix for Elasticsearch search phase execution exception. * OeX_ES-57: Isort fixes. * OeX_ES-57: Creates and apply a decorator to update Request query parameters by Request body data. * OeX_ES-57: Udaptes AggregateCatalogSearchViewSetTests suit and create additional one to test support for both query and body parameters. * OeX_ES-57: Pylint fixes. * OeX_ES-57: Improve naming. * OeX_ES-57: Megre migrations after rebase to edx upstream master. * OeX_ES-57: Pylint fixes after the rebase. * OeX_ES-57: Deletes my own migrations because they are already in the master branch. * OeX_ES-57: Fixed a problem when a list is passing during a Post request. * OeX_ES-57: Adds a new feature when filtering by org and partner fields is can be case insensitive but has keyword behavior. * OeX_ES-57: Adds 'type' fields of Person index to be filtered out. * OeX_ES-57: Adds 'availability' fields of Course and CourseRun index to be filtered out. * OeX_ES-57: Adds 'level_type' fields of Common index to be filtered out. * OeX_ES-57: Adds 'first_enrollable_paid_seat_price' field to be filtered out. * OeX_ES-57: Adds lookup exclude for 'org' field. * OeX_ES-57: Setup for availability and for level_type exect match. I think this is right decision. * OeX_ES-57: feat(Indexes). Adds the ability for some index fields to be available for case-insensitive filtering. * OeX_ES-57: feat(ES search view). Expands fields for filtering so that catalog's queries for the enterprize platform work. Sorted the filter and facet fields alphabetically. * OeX_ES-57: fix(Catalog filter). Adds additional conditions when don't need to filter by default partner. * OeX_ES-57: test(Catalog filter). Adds a new test whet POST search/all query supports empry body payload data. * OeX_ES-57: fix(ES search view). Adds field enrollment_mode as legacy support. * OeX_ES-57: fix(ES filter backend). If fields to filter has 'lower' meaning than make value in lower case for all cases. * OeX_ES-57: fix(cleaning). Pylint polishing * OeX_ES-57: Creates new Match filter backend mixin to be able filter by substring and use 'match' action in es filtering. * OeX_ES-57: Makes 'key' field is ready to filter exact by 'key.raw' and match by substings. * OeX_ES-57: Updates broken tests and add new to test that key field works in filter by substing mode. * OeX_ES-57: Uses LOOKUP_FILTER_MATCH as default lookup for 'key' field. * OeX_ES-57: Adds 'short_description' field to the filter list and uses LOOKUP_FILTER_MATCH as default lookup for the field. * OeX_ES-57: Refuses to es 'match' the filter. Instead, I began to use the 'match phrase' filter. I also made a solution for the filter to work by or if pass an array of values. Co-authored-by: nikolay.borovenskiy <nikolay.borovenskiy@gmail.com>