September 1, 2024
By: Kevin
clojure复杂对象的序列化/反序列化
序列化的方法
clojure的数据类型(数字, 字符串, uuid, 时间, bigint, ratio等)以及自带数据结构比如map, list, vector, set等都有字面意义的字符串表示, 可以很容易的使用clojure的reader从文本读入, 使用pr函数输出为可以被读入的文本表示.
通过reader, clojure自带的edn(Extensible Data Notation)很好的解决了语言自带类型的序列化的问题.
类型上比json丰富的多. 但是还不太够. 常见的问题有:
- Java对象的序列化的问题.
- 二进制格式导入/导出, 减小体积
- 高性能的序列化/反序列化要求
- 除了数据, 函数类型的序列化
- 安全性要求, 需要对导出的数据进行加密
Nippy是一个成熟的, 小体积的, 高性能的, 容易扩展的, 高压缩率的, 支持加密的reader替代品.
支持的类型
{:nil nil
:true true
:false false
:false-boxed (Boolean. false)
:char \ಬ
:str-short "ಬಾ ಇಲ್ಲಿ ಸಂಭವಿಸ"
:str-long (reduce str (range 1024))
:kw :keyword
:kw-ns ::keyword
:sym 'foo
:sym-ns 'foo/bar
:kw-long (keyword (reduce str "_" (range 128)) (reduce str "_" (range 128)))
:sym-long (symbol (reduce str "_" (range 128)) (reduce str "_" (range 128)))
:byte (byte 16)
:short (short 42)
:integer (int 3)
:long (long 3)
:float (float 3.1415926535897932384626433832795)
:double (double 3.1415926535897932384626433832795)
:bigdec (bigdec 3.1415926535897932384626433832795)
:bigint (bigint 31415926535897932384626433832795)
:ratio 22/7
:list (list 1 2 3 4 5 (list 6 7 8 (list 9 10 (list) ())))
:vector [1 2 3 4 5 [6 7 8 [9 10 [[]]]]]
:subvec (subvec [1 2 3 4 5 6 7 8] 2 8)
:map {:a 1 :b 2 :c 3 :d {:e 4 :f {:g 5 :h 6 :i 7 :j {{} {}}}}}
:map-entry (clojure.lang.MapEntry/create "key" "val")
:set #{1 2 3 4 5 #{6 7 8 #{9 10 #{#{}}}}}
:meta (with-meta {:a :A} {:metakey :metaval})
:nested [#{{1 [:a :b] 2 [:c :d] 3 [:e :f]} [#{{[] ()}}] #{:a :b}}
#{{1 [:a :b] 2 [:c :d] 3 [:e :f]} [#{{[] ()}}] #{:a :b}}
[1 [1 2 [1 2 3 [1 2 3 4 [1 2 3 4 5 "ಬಾ ಇಲ್ಲಿ ಸಂಭವಿಸ"] {} #{} [] ()]]]]]
:regex #"^(https?:)?//(www\?|\?)?"
:sorted-set (sorted-set 1 2 3 4 5)
:sorted-map (sorted-map :b 2 :a 1 :d 4 :c 3)
:lazy-seq-empty (map identity ())
:lazy-seq (repeatedly 64 #(do nil))
:queue-empty (into clojure.lang.PersistentQueue/EMPTY [:a :b :c :d :e :f :g])
:queue clojure.lang.PersistentQueue/EMPTY
:uuid (java.util.UUID. 7232453380187312026 -7067939076204274491)
:uri (java.net.URI. "https://clojure.org")
:defrecord (nippy/StressRecord. "data")
:deftype (nippy/StressType. "data")
:bytes (byte-array [(byte 1) (byte 2) (byte 3)])
:objects (object-array [1 "two" {:data "data"}])
:util-date (java.util.Date. 1577884455500)
:sql-date (java.sql.Date. 1577884455500)
:instant (java.time.Instant/parse "2020-01-01T13:14:15.50Z")
:duration (java.time.Duration/ofSeconds 100 100)
:period (java.time.Period/of 1 1 1)
:throwable (Throwable. "Msg")
:exception (Exception. "Msg")
:ex-info (ex-info "Msg" {:data "data"})
:many-longs (vec (repeatedly 512 #(rand-nth (range 10))))
:many-doubles (vec (repeatedly 512 #(double (rand-nth (range 10)))))
:many-strings (vec (repeatedly 512 #(rand-nth ["foo" "bar" "baz" "qux"])))
:many-keywords (vec (repeatedly 512
#(keyword
(rand-nth ["foo" "bar" "baz" "qux" nil])
(rand-nth ["foo" "bar" "baz" "qux" ]))))}
序列化
(def frozen-stress-data (nippy/freeze (nippy/stress-data {})))
=> #<byte[] [B@3253bcf3>
反序列化
(nippy/thaw frozen-stress-data)
=> {:bytes (byte-array [(byte 1) (byte 2) (byte 3)])
:nil nil
:boolean true
<...> }
加密
支持AES128位的加密
(nippy/freeze (nippy/stress-data {}) {:password [:salted "my-password"]}) ; Encrypt
(nippy/thaw <encrypted-data> {:password [:salted "my-password"]}) ; Decrypt
扩展对atom, async/chan 的支持
复杂结构的规则扩展, 可以自定义规则.
(require '[clojure.core.async :refer [chan]])
(require '[taoensso.nippy :as nippy])
(defrecord MyType [data c atm])
(nippy/extend-freeze MyType :my-type/foo
[x data-output]
(.writeUTF data-output (str (-> x
(dissoc :c)
(update :atm #(deref %))))))
(nippy/extend-thaw :my-type/foo ; Same type id
[data-input]
(-> (.readUTF data-input)
(read-string)
(update :atm #(atom %))
(#(assoc % :c (chan)))
(map->MyType)))
(nippy/thaw (nippy/freeze (->MyType "Joe" (chan) (atom {:a "a" :b 100}))))
扩展对函数序列化的支持
(ns example
(:require [taoensso.nippy :as nippy]
[com.rpl.nippy-serializable-fn]))
(defn my-fn [a b c]
(+ a b c))
(def thawed-myfn
(nippy/thaw! (nippy/freeze! my-fn)))
(thawed-myfn 1 2 3) ; 6
(let [x 10
afn (fn [a b c] (+ a b c x))
fn-bytes (nippy/freeze! afn)
thawed-fn (nippy/thaw! fn-bytes)]
(thawed-fn 1 2 3) ; 16
)