MHRD Projects Gitlab

Skip to content
  • Feanil Patel's avatar
    Feanil/es7 upgrade (#2870) · 17891023
    Feanil Patel authored
    
    
    * OeX_ES-33: Use django-es-dsl into search model methods. Set ordering by id for Program model.
    
    * OeX_ES-33: Uses django-es-dsl for course runs views.
    
    * OeX_ES-33: Uses django-es-dsl for catalog queries views.
    
    * OeX_ES-33: Removes haystack and uses django-es-dsl for catalogs models.
    
    * OeX_ES-33: For clean query utils handle the case when query string contains a whitespase
    
    * OeX_ES-33: Adds customs text analyzers. Particularly 'html_strip', 'synonym_text'.
    
    * OeX_ES-33: Adds ES course document.
    
    * OeX_ES-33: Adds ES course run document.
    
    * OeX_ES-33: Adds ES person document.
    
    * OeX_ES-33: Adds ES program document.
    
    * OeX_ES-33: Common indexes logic.
    
    * OeX_ES-33: determs Elasticsearch documents package
    
    * OeX_ES-33: Updates course runs view test module
    
    * OeX_ES-33: Updates programs view test module
    
    * OeX_ES-33: Updates search view test module
    
    * OeX_ES-33: Updates tests utils module from core app to cut off haystack.
    
    * OeX_ES-33: Adds a fix to SearchQuerySetWrapper to provide es document by index.
    
    * OeX_ES-33: Fix for CourseRun model to be able search via '(*)'.
    
    * OeX_ES-33: Updates test_models test module of course_metadata app.
    
    * OeX_ES-33: Updates test_signals test module of course_metadata app.
    
    * OeX_ES-33: For es PersonDocument rebuilt position field generation.
    
    * OeX_ES-33: For es ProgramDocument fix for prepare sit types. And subject_uuids, authoring_organization_uuids, staff_uuids has to be keyword instead of text.
    
    * OeX_ES-33: Updates test_serializers test module of api app.
    
    * OeX_ES-33: Creates generic serializers mixins to handle es response. ModelObjectDocumentSerializerMixin and DocumentDSLSerializerMixin.
    
    * OeX_ES-33: Describes es serializers package interface.
    
    * OeX_ES-33: Adds serializer for course document for elasticsearch response.
    
    * OeX_ES-33: Adds serializer for course run document for elasticsearch response.
    
    * OeX_ES-33: Adds serializer for person document for elasticsearch response.
    
    * OeX_ES-33: Adds serializer for program document for elasticsearch response.
    
    * OeX_ES-33: Adds constants for search indexes.
    
    * OeX_ES-33: Adds custom elasticsearch real time signal processor.
    
    * OeX_ES-33: Changes template text file name to be able use old es text templates.
    
    * OeX_ES-33: Mixin class for supporting faceting for into elastissearch on an API View.
    
    * OeX_ES-33: Fix for elasticsearch boost config. Cause negative  is not supported anymore.
    
    * OeX_ES-33: In order to es7 synonyms have to be list of string with comma separated values.
    
    * OeX_ES-33: Cut off and rebuilt es for main conftest file to be able get es connection and create/deletes es indexes.
    
    * OeX_ES-33: Updates ElasticsearchTestMixin.
    
    * OeX_ES-33: Renames elasticsearch dsl default connection fixture.
    
    * OeX_ES-33: Updates test view mixins.
    
    * OeX_ES-33: Moves search engine serializers to search_indexes module. Cut off haystack from api/serializers.
    
    * OeX_ES-33: Fixes test serializers module.
    
    * OeX_ES-33: Refactores search indexes module serializers.
    
    * OeX_ES-33: Updates test search module.
    
    * OeX_ES-33: Updates all es search viewsets.
    
    * OeX_ES-33: Updates test create index. Deletes test alias exists cause django es dsl has own manage.py commands to manage indexes.
    
    * OeX_ES-33: adds aliases for es indexes.
    
    * OeX_ES-33: Puts changes to function_score which are relevant to es7 requirements.
    
    * OeX_ES-33: Updates test boosting module of edx haystack extensions app.
    
    * OeX_ES-33: Extends Document class of boosting search
    
    * OeX_ES-33: For ProgramDocument changes fiels type for  attribute.
    
    * OeX_ES-33: Deletes haystacks serializers from aggregate facet search serializers.
    
    * OeX_ES-33: Updates edx catalog extensions serializers.
    
    * OeX_ES-33: Updates faceted search fields for  attribute
    
    * OeX_ES-33: Updates edx catalog extensions test views tests.
    
    * OeX_ES-33: Adds custom FacetedFieldSearchFilterBackend to Facet view mixin.
    
    * OeX_ES-33: Modifies distinct count module. Due to architectural aspects of elasticsearch dsl removed old haystack QueryBackend and moved the logic to the customized Search class.
    
    * OeX_ES-33: Uses  constant from search indexes package
    
    * OeX_ES-33: Adds a module of django elasticsearch dsl drf specific serializers.
    
    * OeX_ES-33: Refactoring due removing custom response class to separete module.
    
    * OeX_ES-33: Adds module with extended elasticsearch-dsl responses.
    
    * OeX_ES-33: Adds module with extended elasticsearch-dsl Search classes.
    
    * OeX_ES-33: Adds module with extended django-elasticsearch-dsl-drf viewsets. Also implements BaseElasticsearchDocumentViewSet.
    
    * OeX_ES-33: Adds module with extended django-elasticsearch-dsl-drf backends.
    
    * OeX_ES-33: Deletes a module with hastack implementation of es indexes. It's really so sad. but time is up for them.
    
    * OeX_ES-33: Renames edx_haystack_extensions application name on a new one edx_elasticsearch_dsl_extensions.
    
    * OeX_ES-33: Handles the case where query value already coming in quoted.
    
    * OeX_ES-33: Adds edge_ngram_completion custom analyzer.
    
    * OeX_ES-33: Specifies addition settings for all es indexes.
    
    * OeX_ES-33: Updates course-discovery settings.
    
    * OeX_ES-33: Deletes not related now drf filters and their all tests.
    
    * OeX_ES-33: Elasticsearch dsl add xdist suffix to index name works now.
    
    * OeX_ES-33: Moves common filter backends(MultiMatchSearchFilterBackend, DefaultOrderingFilterBackend) to BaseElasticsearchDocumentViewSet.
    
    * OeX_ES-33: Fix for multi document serializer mixin to be able work in mulri processes mode.
    
    * OeX_ES-33: Uses ELASTICSEARCH_DSL_SIGNAL_PROCESSOR instead of HAYSTACK_SIGNAL_PROCESSOR for UpdateIndexTests.
    
    * OeX_ES-33: Deletes  from django INSTALLED_APPS.
    
    * OeX_ES-33: Refactoring. For CourseRunTests test suit use elasticsearch_dsl_default_connection fixture instead of manage.py commands to create es indexes.
    
    * OeX_ES-33: Updates Tracis CI docker-compose yaml in es7 perspective.
    
    * OeX_ES-33: Fix for test course runs module.
    
    * Revert "OeX_ES-33: Fix for test course runs module."
    
    This reverts commit a06bb4ccded65756ac08665ac77c147e5ef1a521.
    
    * OeX_ES-33: Adds get_latest_by field to Program model explicitly.
    
    * OeX_ES-33: To be 100% sure, deletes all es indexes after a test session.
    
    * OeX_ES-33: Just in case deletes all index before test  to have clean conditions.
    
    * OeX_ES-33: Updates ELASTICSEARCH_INDEX_NAMES in multiprocesses test execution mode too. The names have to be updated.
    
    * OeX_ES-33: Adds additional request value due travis add one request for own purposes.
    
    * OeX_ES-33: Style check. Reordes imports.
    
    * OeX_ES-33: Modefies paid_seat_enrollment_end boost to still to be able to penalize course runs without enrollable, paid seats.
    
    * OeX_ES-33: Adds ELASTICSEARCH_DSL_INDEX_RETENTION_LIMIT to base settings.
    
    * OeX_ES-33: Updates command  to be able use it when there are four indexes and no haystack lib.
    
    * OeX_ES-33: Updates tests for the command .
    
    * OeX_ES-33: Deletes senseless alias  from all es indexes.
    
    * OeX_ES-33: Update  index utils.
    
    * OeX_ES-33: Rebuilds clear_es)indexes fixtute to be able delete all indexes after a test session by alias prefix.
    
    * OeX_ES-33: Adds threshold to test signals module cause Travis sometimes adds own db requests.
    
    * OeX_ES-33: Rebuilds command . Now the command bases on django-es-dsl.
    
    * OeX_ES-33: Deletes useless test mixins.
    
    * OeX_ES-33: Updates tests for  command.
    
    * OeX_ES-33: Impemented  feature in context index creation and updation. Add a new commands  to search_index genaral django-elasticsearch-dsl managments.
    
    * OeX_ES-33: Removes  action handler from overwritten search_index command.
    
    * OeX_ES-33: Style check fixes.
    
    * OeX_ES-33: Upgrades course-discovery requirements. Deletes django-hastack and drf-hastack from requirements and service at all. Goodbye, you would be great tools and fighters. Thanks for all! We will miss you.
    
    * OeX_ES-33: Adds additional style check fixes.
    
    * OeX_ES-33: Updates test boosting module to be able put back the case when course runs with expired paid seats are penalized.
    
    * OeX_ES-33: Pins pacing type for CourseRun model factory during test_typeahead_org_course_runs_come_up_first. And add message about an index creating for search_index --create command.
    
    * OeX_ES-33: Adds  method to Elasticsearch global utils.
    
    * OeX_ES-33: Removes logic about  form overwtitten  command.
    
    * OeX_ES-33: Puts back  command. Now the sanity check feature works as before.
    
    * OeX_ES-33: Updates update index tests cause new(old) command was developed.
    
    * OeX_ES-33: Fix import orders.
    
    * OeX_ES-33: Updates README. Adds new discription about base django-elasticsearch-dsl commands. Adds info about  command.
    
    * OeX_ES-33: Refactoring for es documents. Moves all common fields to Base document.
    
    * OeX_ES-33: Code polishing.
    
    * OeX_ES-33: Fix for CourseRunViewSetTests test suit.
    
    * OeX_ES-33: Adds migrations. Mostly due to changing Program's default ordering, but also looks like some parler translation model field changes happened.
    
    * OeX_ES-33: After an index creation assigns to internal index name alias name.
    
    * OeX_ES-33: Deletes  filter tokenizer from synonym text analyzer.
    
    * OeX_ES-33: Index matching by alias not by index name as was before.
    
    * OeX_ES-33: Extends  and  to be able delete indexes by index name cause delete by alias is impossible in es.
    
    * OeX_ES-33: Improves, fixes and simplifies  command.
    
    * OeX_ES-33: Updates tests for  and  manage.py commands.
    
    * OeX_ES-33: Isort fix.
    
    * OeX_ES-33: Pylint fixes.
    
    * OeX_ES-33: Code polishing after a code review session.
    
    * OeX_ES-33: Fix for  aggregation.
    
    * OeX_ES-33: Merged migrations
    
    * OeX_ES-33: Fix tests after rebasing on upstream master
    
    * OeX_ES-33: Deletes threshold for test_draft_does_not_collide_with_draft test.
    
    * OeX_ES-33: Pull data from ELASTICSEARCH_CLUSTER_URL instead of ELASTICSEARCH_URL to make cutover easier.
    
    * Revert "OeX_ES-33: Deletes threshold for test_draft_does_not_collide_with_draft test."
    
    This reverts commit a24af04fdd4e800573d7c4e58b16502c9184812e.
    
    * OeX_ES-33: Uses FuzzyInt instead of threshold value for 'test_collision_does_not_include_drafts' test.
    
    * OeX_ES-57: Reworks 'clean_query' function. Deletes lowercase transformation there since now it does elasticsearch itself.
    
    * OeX_ES-57: Adds new tests to cover search by key for all-list view.
    
    * OeX_ES-57: Adds search by 'key' and 'ord' fields.
    
    * OeX_ES-57: Enables 'analyze_wildcard' option for query_string type of search.
    
    * OeX_ES-57: Code polishing.
    
    * OeX_ES-57: Fixes problem when query value contains ':' character. To fix it override split lookup name for base search filter backend. New character is ':='.
    
    * OeX_ES-57: Adds for each request which is handled by django-es-dsl-drf view 'collapse' feature. Reason: attemp to fight 'duplicate issue' during es search.
    
    * OeX_ES-57: Refactoring. Moves 'prepare_authoring_organization_uuids' method to base document.
    
    * OeX_ES-57: Deletes key field from es templates. This is no longer necessary. Takes only redundancy.
    
    * OeX_ES-57: Extends multi match search firlds. Now we are able to search by key separately.
    
    * OeX_ES-57: Refactors es search views to add 'authoring_organization_uuids' fiels to filter fields. Also adds LimitedAggregateSearchView view filter backends.
    
    * OeX_ES-57: Extends LimitedAggregateSearchViewSetTests test suit to test the search results include the authoring_organization_uuids for each document.
    
    * OeX_ES-57: Adds FilteringFilterBackend for LimitedAggregateSearchView view.
    
    * OeX_ES-57: Isort fix.
    
    * OeX_ES-57: Pylint fixes.
    
    * OeX_ES-57: Adds additional lost fields to Course document.
    
    * OeX_ES-57: Changes multi match strategy. Now let's try 'phrase' match.
    
    * OeX_ES-57: Moves the logic about datetime serializetion to separete Mixin and use it.
    
    * OeX_ES-57: Add a new test to catalog suit to check courses with different catalog queries but the same meaning
    
    * OeX_ES-57: Add a new test to catalog suit to check courses with time range query.
    
    * OeX_ES-57: Add a new test to catalog suit to check courses with subjects and negative query.
    
    * OeX_ES-57: Uses for tests based on key the key which was used in a live debug session.
    
    * OeX_ES-57: Fix for Multi Document Serializer to be able use its own fields for an each specific serializer.
    
    * OeX_ES-57: Adds test to be sure that that draft objects are not indexed when run command to populate index.
    
    * OeX_ES-57: Adds test to be sure that that draft objects are not indexed when es indexes are being updated via django's signals.
    
    * OeX_ES-57: ES documents has not use django default model manager. Has to use particular one determined of the application.
    
    * OeX_ES-57: Fix. Multi document serializer has to define patch necessity when 'field' attr exist and not none.
    
    * OeX_ES-57: Do not use separete fields for AggregateSearchSerializer.
    
    * OeX_ES-57: Adds index update handler via django's signal when need to prevent indexing in the case when object is Draft.
    
    * OeX_ES-57: Fix import orders.
    
    * OeX_ES-57: Semgrep django style fixes.
    
    * OeX_ES-57: Raises IndexForbiddenException if es index is impossible.
    
    * OeX_ES-57: Pylint fixes.
    
    * OeX_ES-57: Serialize datetime fix for test serializers.
    
    * OeX_ES-57: Cause Draft object indexing is disabled now I have to change queries numbers for test_draft_does_not_collide_with_draft.
    
    * OeX_ES-57: Refuses collapsing as additional parametr for each es query.
    
    * OeX_ES-57: Puts ELASTICSEARCH_DSL_LOAD_PER_QUERY to course discovery settings.
    
    * OeX_ES-57: Adds default pagination to Extended es dsl Search class.
    
    * OeX_ES-57: Refactors common es document module.
    
    * OeX_ES-57: Adds test suit TestSearchPagination for es documents.
    
    * OeX_ES-57: Isort quality fixes.
    
    * OeX_ES-57: Bug fix. ES count request does not support paginated payload. So do not insert from and size to the request.
    
    * Change catalog test casing.
    
    - Org names can be a different case than the search query says.
    - The same is probably also true for the subjects so adding that to the test as well.
    
    * OeX_ES-57: Make for course based documents org field text and keyword types.
    
    * OeX_ES-57: Changes facet field for 'org' on 'org.raw'
    
    * OeX_ES-57: Deletes all courses to make clear environment for test_courses_with_subjects_and_negative_query test.
    
    * OeX_ES-57: Defines a module for specific custom exceptions for Elasticsearch.
    
    * OeX_ES-57: Fix for Elasticsearch search phase execution exception.
    
    * OeX_ES-57: Adds a django mixin to catch invalid Elasticsearch query string exception.
    
    * OeX_ES-57: Adds tests for the fix for Elasticsearch search phase execution exception.
    
    * OeX_ES-57: Isort fixes.
    
    * OeX_ES-57: Creates and apply a decorator to update Request query parameters by Request body data.
    
    * OeX_ES-57: Udaptes AggregateCatalogSearchViewSetTests suit and create additional one to test support for both query and body parameters.
    
    * OeX_ES-57: Pylint fixes.
    
    * OeX_ES-57: Improve naming.
    
    * OeX_ES-57: Megre migrations after rebase to edx upstream master.
    
    * OeX_ES-57: Pylint fixes after the rebase.
    
    * OeX_ES-57: Deletes my own migrations because they are already in the master branch.
    
    * OeX_ES-57: Fixed a problem when a list is passing during a Post request.
    
    * OeX_ES-57: Adds a new feature when filtering by org and partner fields is can be case insensitive but has keyword behavior.
    
    * OeX_ES-57: Adds 'type' fields of Person index to be filtered out.
    
    * OeX_ES-57: Adds 'availability' fields of Course and CourseRun index to be filtered out.
    
    * OeX_ES-57: Adds 'level_type' fields of Common index to be filtered out.
    
    * OeX_ES-57: Adds 'first_enrollable_paid_seat_price' field to be filtered out.
    
    * OeX_ES-57: Adds lookup exclude for 'org' field.
    
    * OeX_ES-57: Setup for availability and for level_type exect match. I think this is right decision.
    
    * OeX_ES-57: feat(Indexes). Adds the ability for some index fields to be available for case-insensitive filtering.
    
    * OeX_ES-57: feat(ES search view). Expands fields for filtering so that catalog's queries for the enterprize platform work. Sorted the filter and facet fields alphabetically.
    
    * OeX_ES-57: fix(Catalog filter). Adds additional conditions when don't need to filter by default partner.
    
    * OeX_ES-57: test(Catalog filter). Adds a new test whet POST search/all query supports empry body payload data.
    
    * OeX_ES-57: fix(ES search view). Adds field enrollment_mode as legacy support.
    
    * OeX_ES-57: fix(ES filter backend). If fields to filter has 'lower' meaning than make value in lower case for all cases.
    
    * OeX_ES-57: fix(cleaning). Pylint polishing
    
    * OeX_ES-57: Creates new Match filter backend mixin to be able filter by substring and use 'match' action in es filtering.
    
    * OeX_ES-57: Makes 'key' field is ready to filter exact by 'key.raw' and match by substings.
    
    * OeX_ES-57: Updates broken tests and add new to test that key field works in filter by substing mode.
    
    * OeX_ES-57: Uses LOOKUP_FILTER_MATCH as default lookup for 'key' field.
    
    * OeX_ES-57: Adds 'short_description' field to the filter list and uses LOOKUP_FILTER_MATCH as default lookup for the field.
    
    * OeX_ES-57: Refuses to es 'match' the filter. Instead, I began to use the 'match phrase' filter. I also made a solution for the filter to work by or if pass an array of values.
    
    Co-authored-by: default avatarnikolay.borovenskiy <nikolay.borovenskiy@gmail.com>
    17891023