Codeberg

  • 4 posts
  • 27 comments
Joined 3 years ago
Cake day: July 12th, 2023
Programming@programming.devbytapdattl@lemmy.world
5 months

Best Practices for Encrypted Search

Task

I’m working on my final project for school, we are supposed to make a web app of our choosing and there has to be specific features in it. One of it is all data must be encrypted, and the other is that we have to have a search functionality. My app (A customer support framework) has a ticket functionality where customers can submit help request tickets, the contents of these tickets need to be encrypted at rest, at the same time admins need to be able to search contents of tickets.

Current Plan

My current plan is to store an AES-256 encrypted copy of the message message.content to meet the encrypted requirement, and also store a tokenized and hashed version of the message message.hashed to meet the searchability requirement.

The tokenization/hashing method will be:

  • strip the message to alphanumeric + whitespace ([a-zA-Z0-9 ])
  • tokenize by splitting the message by whitespace,
  • SHA-256 each token,
  • rejoin all the hashed tokens into a space seperated string and stored in the message.hashed field.

Thus this is a test string becomes <hash of this> <hash of is> <hash of a> <hash of test> <hash of string>

When the user searches their search string goes through all of the steps in the tokenization/hashing method, then we query the message table for message.hashed LIKE %%<hashed string>%% and if my thinking is right, we should be able to find it.

Concerns

  • Statistical analysis of hashed tokens
    • I really don’t see a way around this, to make the string searchable the hashing needs to be predictable.
  • message.hashed field could potentially be huge, if each word is getting a SHA256 hash, a large message could result in a very large hash string
    • maybe we just store the last 4 of the hash?
      • This would increase collisions, but the likelihood of multiple last 4’s colliding in a given search string should be pretty dang small, and any collisions would likely not be valid language.
      • Would this help with the statistical analysis concern? Increasing collisions would decrease the effectiveness of statistical analysis. It would be a performance hit, but after returning all matches against the hashes I could decrypt the message.content data and search the raw search query against the unencrypted text and remove any incorrect returns caused by collisions.

I’m interested in hearing everyone’s thoughts, am I being logical in my reasoning?

8
    • tapdattl@lemmy.worldtoSelfhosted@lemmy.world•Ideal Business Stack?English
      1 year

      FreeIPA and Keycloak will give you directory management (LDAP and Kerberos), identity management, and single-sign on (OIDC and SAML) which if all your computers are running Linux as well, will give you centralized management of users.

      You can then set other FOSS business management/productivity applications like NextCloud, Oodoo, Seafile, OnlyOffice, LibreOffice, CryptPad, etc. To use Keycloak as its authentication mechanism.

      A lot of this will depend on what kind of work the business does.

      You’ll also want to look into log management and SEIM for security monitoring, Wazuh, Graylog, and others. This is especially true if the business has any data compliancy responsibilities in the country this is in.

    • tapdattl@lemmy.worldtoGolang@programming.dev•*Permanently Deleted*
      1 year

      there’s no explanation of what this is supposed to do.

      Totally right, sorry about that, I’ll update the Github, but it brief this is a library that’s supposed to help a developer set up a Role Based Access Control system for an API for web service. Role Based Access Control is a method of access control whereby (And this is my very beginner’s understanding of it) users are assigned roles, and these roles are in turn issued different permissions based off what that role is supposed to have access to. When checking if a user is authorized access to a certain resource, the roles assigned to them are checked for the permissions needed for the resource. If they have permission then they are granted access to the resource, otherwise they are denied access.

      This library manages roles, permissions assigned to roles, and checking of permissions against roles via an http middleware.

      Then, there’s no main function. Where’s the entry point? This is a bit where I’m doubting myself now. Maybe go has changed, but when I was writing it, it requires a main function to even run.

      Well, this is supposed to be a library that’s used by other people, so it has no main function itself, rather it’s called by other people