Middlewares as Go Plugins PoC [DO NOT MERGE] #6260
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
API Changes --- prev.txt 2024-05-07 15:51:36.936092544 +0000
+++ current.txt 2024-05-07 15:51:33.736044656 +0000
@@ -1806,11 +1806,9 @@
Method string `bson:"method" json:"method"`
}
-type TykEvent = event.Event
- TykEvent is an alias maintained for backwards compatibility.
+type TykEvent string // A type so we can ENUM event types easily, e.g. EventQuotaExceeded
-type TykEventHandlerName = event.HandlerName
- TykEventHandlerName is an alias maintained for backwards compatibility.
+type TykEventHandlerName string // A type for handler codes in API definitions
type UDGGlobalHeader struct {
Key string `bson:"key" json:"key"`
@@ -1949,29 +1947,6 @@
ProxyOnError bool `bson:"proxy_on_error" json:"proxy_on_error"`
}
-type WebHookHandlerConf struct {
- // Disabled enables/disables this webhook.
- Disabled bool `bson:"disabled" json:"disabled"`
- // ID optional ID of the webhook, to be used in pro mode.
- ID string `bson:"id" json:"id"`
- // Name is the name of webhook.
- Name string `bson:"name" json:"name"`
- // The method to use for the webhook.
- Method string `bson:"method" json:"method"`
- // The target path on which to send the request.
- TargetPath string `bson:"target_path" json:"target_path"`
- // The template to load in order to format the request.
- TemplatePath string `bson:"template_path" json:"template_path"`
- // Headers to set when firing the webhook.
- HeaderList map[string]string `bson:"header_map" json:"header_map"`
- // The cool-down for the event so it does not trigger again (in seconds).
- EventTimeout int64 `bson:"event_timeout" json:"event_timeout"`
-}
- WebHookHandlerConf holds configuration related to webhook event handler.
-
-func (w *WebHookHandlerConf) Scan(in any) error
- Scan scans WebHookHandlerConf from `any` in.
-
# Package: ./apidef/adapter
package adapter // import "github.com/TykTechnologies/tyk/apidef/adapter"
@@ -2933,43 +2908,6 @@
func (et *EnforceTimeout) Fill(meta apidef.HardTimeoutMeta)
Fill fills *EnforceTimeout from apidef.HardTimeoutMeta.
-type Event struct {
- // Enabled enables the event handler.
- Enabled bool `json:"enabled" bson:"enabled"`
- // Type specifies the TykEvent that should trigger the event handler.
- Type event.Event `json:"type" bson:"type"`
- // Action specifies the action to be taken on the event trigger.
- Action event.Action `json:"action" bson:"action"`
- // ID is the ID of event handler in storage.
- ID string `json:"id,omitempty" bson:"id,omitempty"`
- // Name is the name of event handler
- Name string `json:"name,omitempty" bson:"name,omitempty"`
-
- Webhook WebhookEvent `bson:"-" json:"-"`
-}
- Event holds information about individual event to be configured on the API.
-
-func (e *Event) GetWebhookConf() (map[string]interface{}, error)
- GetWebhookConf converts Event.WebhookEvent to map[string]interface{}
- with apidef.WebHookHandlerConf structure for classic API definition
- compatibility.
-
-func (e *Event) MarshalJSON() ([]byte, error)
- MarshalJSON marshals Event as per Tyk OAS API definition contract.
-
-func (e *Event) UnmarshalJSON(in []byte) error
- UnmarshalJSON unmarshals Event as per Tyk OAS API definition contract.
-
-type Events []Event
- Events holds the list of events to be processed for the API.
-
-func (e *Events) ExtractTo(api *apidef.APIDefinition)
- ExtractTo extracts events to apidef.APIDefinition.
-
-func (e *Events) Fill(api apidef.APIDefinition)
- Fill fills Events from classic API definition. Currently only webhook events
- are supported.
-
type ExternalOAuth struct {
Enabled bool `bson:"enabled" json:"enabled"` // required
AuthSources `bson:",inline" json:",inline"`
@@ -3889,11 +3827,6 @@
//
// Tyk classic API definition: `detailed_tracing`
DetailedTracing *DetailedTracing `bson:"detailedTracing,omitempty" json:"detailedTracing,omitempty"`
-
- // Events contains the configuration related to Tyk Events.
- //
- // Tyk classic API definition: `event_handlers`
- Events Events `bson:"events,omitempty" json:"events,omitempty"`
}
Server contains the configuration that sets Tyk up to receive requests from
the client applications.
@@ -4412,15 +4345,6 @@
It is implemented to facilitate a smooth migration from deprecated fields
that were previously used to represent the same data.
-type WebhookEvent struct {
- URL string `json:"url" bson:"url"`
- Method string `json:"method" bson:"method"`
- Timeout int64 `json:"timeout" bson:"timeout"`
- BodyTemplate string `json:"bodyTemplate,omitempty" bson:"bodyTemplate,omitempty"`
- Headers map[string]string `json:"headers,omitempty" bson:"headers,omitempty"`
-}
- WebhookEvent stores the core information about a webhook event.
-
type XTykAPIGateway struct {
// Info contains the main metadata for the API definition.
Info Info `bson:"info" json:"info"` // required
@@ -5375,7 +5299,6 @@
func (c *Config) LoadIgnoredIPs()
func (c *Config) SetEventTriggers(eventTriggers map[apidef.TykEvent][]TykEventHandler)
- SetEventTriggers sets events for backwards compatibility
func (c *Config) StoreAnalytics(ip string) bool
@@ -6870,40 +6793,26 @@
other types. This is a go <1.13 cludge.
const (
- // EventQuotaExceeded is an alias maintained for backwards compatibility.
- EventQuotaExceeded = event.QuotaExceeded
- // EventRateLimitExceeded is an alias maintained for backwards compatibility.
- EventRateLimitExceeded = event.RateLimitExceeded
-
- // EventAuthFailure is an alias maintained for backwards compatibility.
- EventAuthFailure = event.AuthFailure
- // EventKeyExpired is an alias maintained for backwards compatibility.
- EventKeyExpired = event.KeyExpired
- // EventVersionFailure is an alias maintained for backwards compatibility.
- EventVersionFailure = event.VersionFailure
- // EventOrgQuotaExceeded is an alias maintained for backwards compatibility.
- EventOrgQuotaExceeded = event.OrgQuotaExceeded
- // EventOrgRateLimitExceeded is an alias maintained for backwards compatibility.
- EventOrgRateLimitExceeded = event.OrgRateLimitExceeded
- // EventTriggerExceeded is an alias maintained for backwards compatibility.
- EventTriggerExceeded = event.TriggerExceeded
- // EventBreakerTriggered is an alias maintained for backwards compatibility.
- EventBreakerTriggered = event.BreakerTriggered
- // EventBreakerTripped is an alias maintained for backwards compatibility.
- EventBreakerTripped = event.BreakerTripped
- // EventBreakerReset is an alias maintained for backwards compatibility.
- EventBreakerReset = event.BreakerReset
- // EventHOSTDOWN is an alias maintained for backwards compatibility.
- EventHOSTDOWN = event.HostDown
- // EventHOSTUP is an alias maintained for backwards compatibility.
- EventHOSTUP = event.HostUp
- // EventTokenCreated is an alias maintained for backwards compatibility.
- EventTokenCreated = event.TokenCreated
- // EventTokenUpdated is an alias maintained for backwards compatibility.
- EventTokenUpdated = event.TokenUpdated
- // EventTokenDeleted is an alias maintained for backwards compatibility.
- EventTokenDeleted = event.TokenDeleted
+ EventQuotaExceeded apidef.TykEvent = "QuotaExceeded"
+ EventRateLimitExceeded apidef.TykEvent = "RatelimitExceeded"
+ EventAuthFailure apidef.TykEvent = "AuthFailure"
+ EventKeyExpired apidef.TykEvent = "KeyExpired"
+ EventVersionFailure apidef.TykEvent = "VersionFailure"
+ EventOrgQuotaExceeded apidef.TykEvent = "OrgQuotaExceeded"
+ EventOrgRateLimitExceeded apidef.TykEvent = "OrgRateLimitExceeded"
+ EventTriggerExceeded apidef.TykEvent = "TriggerExceeded"
+ EventBreakerTriggered apidef.TykEvent = "BreakerTriggered"
+ EventBreakerTripped apidef.TykEvent = "BreakerTripped"
+ EventBreakerReset apidef.TykEvent = "BreakerReset"
+ EventHOSTDOWN apidef.TykEvent = "HostDown"
+ EventHOSTUP apidef.TykEvent = "HostUp"
+ EventTokenCreated apidef.TykEvent = "TokenCreated"
+ EventTokenUpdated apidef.TykEvent = "TokenUpdated"
+ EventTokenDeleted apidef.TykEvent = "TokenDeleted"
)
+ Register new event types here, the string is the code used to hook at the
+ Api Deifnititon JSON/BSON level
+
const (
MsgAuthFieldMissing = "Authorization field missing"
MsgApiAccessDisallowed = "Access to this API has been disallowed"
@@ -6933,17 +6842,6 @@
DEFAULT_ORG_SESSION_EXPIRATION = int64(604800)
)
const (
- ErrAuthAuthorizationFieldMissing = "auth.auth_field_missing"
- ErrAuthKeyNotFound = "auth.key_not_found"
- ErrAuthCertNotFound = "auth.cert_not_found"
- ErrAuthCertExpired = "auth.cert_expired"
- ErrAuthKeyIsInvalid = "auth.key_is_invalid"
-
- MsgNonExistentKey = "Attempted access with non-existent key."
- MsgNonExistentCert = "Attempted access with non-existent cert."
- MsgInvalidKey = "Attempted access with invalid key."
-)
-const (
KID = "kid"
SUB = "sub"
HMACSign = "hmac"
@@ -6991,18 +6889,11 @@
const CoProcessDefaultKeyPrefix = "coprocess-data:"
CoProcessDefaultKeyPrefix is used as a key prefix for this CP.
-const EH_CoProcessHandler = event.CoProcessHandler
- EH_CoProcessHandler is used for event system, maintained here for backwards
- compatibility.
-
-const EH_JSVMHandler = event.JSVMHandler
- EH_JSVMHandler is aliased for backwards compatibility.
+const EH_CoProcessHandler apidef.TykEventHandlerName = "cp_dynamic_handler"
+ Constant for event system.
-const (
- // EH_LogHandler is an alias maintained for backwards compatibility.
- // It is used to register log handler on an event.
- EH_LogHandler = event.LogHandler
-)
+const EH_JSVMHandler apidef.TykEventHandlerName = "eh_dynamic_handler"
+const EH_LogHandler apidef.TykEventHandlerName = "eh_log_handler"
The name for event handlers as defined in the API Definition JSON/BSON
format
@@ -7092,6 +6983,14 @@
func CreateStandardPolicy() *user.Policy
func CreateStandardSession() *user.SessionState
+func CtxCheckLimits(r *http.Request) bool
+func CtxGetData(r *http.Request) map[string]interface{}
+ exported version
+
+func CtxGetSession(r *http.Request) *user.SessionState
+func CtxSetData(r *http.Request, m map[string]interface{})
+func CtxSetSession(r *http.Request, s *user.SessionState, scheduleUpdate bool, hashKey bool)
+func CtxSetSpanAttributes(r *http.Request, mwName string, attrs ...otel.SpanAttribute)
func DoCoprocessReload()
func DurationToMillisecond(d time.Duration) float64
func EncodeRequestToEvent(r *http.Request) string
@@ -7099,6 +6998,7 @@
it to base64 and store it in an Event object
func EnsureTransport(host, protocol string) string
+func ErrorAndStatusCode(errType string) (error, int)
func GenerateTestBinaryData() (buf *bytes.Buffer)
func GetAccessDefinitionByAPIIDOrSession(currentSession *user.SessionState, api *APISpec) (accessDef *user.AccessDefinition, allowanceScope string, err error)
func GetTLSClient(cert *tls.Certificate, caCert []byte) *http.Client
@@ -7116,6 +7016,7 @@
will decode request body as json to map[string]string and adds the key/value
pairs in r.Form.
+func LoadAuthKeyMW() func(base *BaseMiddleware) TykMiddleware
func LoadPoliciesFromDir(dir string) (map[string]user.Policy, error)
func LoadPoliciesFromFile(filePath string) (map[string]user.Policy, error)
func LoopingUrl(host string) string
@@ -7148,9 +7049,11 @@
PythonSetEnv sets PYTHONPATH, it's called before initializing the
interpreter.
+func ReportHealthValue(spec *APISpec, counter HealthPrefix, value string)
func RevokeAllTokens(storage ExtendedOsinStorageInterface, clientId, clientSecret string) (int, []string, error)
func RevokeToken(storage ExtendedOsinStorageInterface, token, tokenTypeHint string)
func Start()
+func StripBearer(token string) string
func TestReq(t testing.TB, method, urlStr string, body interface{}) *http.Request
func TestReqBody(t testing.TB, body interface{}) io.Reader
func TykGetData(CKey *C.char) *C.char
@@ -7320,16 +7223,6 @@
ProcessRequest will run any checks on the request on the way through the
system, return an error to have the chain fail
-type AuthKey struct {
- *BaseMiddleware
-}
- KeyExists will check if the key being used to access the API is in the
- request data, and then if the key is in the storage engine
-
-func (k *AuthKey) Name() string
-
-func (k *AuthKey) ProcessRequest(_ http.ResponseWriter, r *http.Request, _ interface{}) (error, int)
-
type BaseExtractor struct {
Config *apidef.MiddlewareIdExtractor
IDExtractorConfig apidef.IDExtractorConfig
@@ -7390,6 +7283,8 @@
FireEvent is added to the BaseMiddleware object so it is available across
the entire stack
+func (t *BaseMiddleware) GetAuthToken(authType string, r *http.Request) (string, apidef.AuthConfig)
+
func (t *BaseMiddleware) Init()
func (t *BaseMiddleware) Logger() (logger *logrus.Entry)
@@ -7940,6 +7835,8 @@
func (gw *Gateway) FireSystemEvent(name apidef.TykEvent, meta interface{})
+func (gw *Gateway) GenerateToken(orgID, keyID string, customHashKeyFunction ...string) string
+
func (gw *Gateway) GetApiSpecsFromAccessRights(sess *user.SessionState) []*APISpec
GetApiSpecsFromAccessRights from the session.AccessRights returns the
collection of api specs
@@ -7980,12 +7877,16 @@
func (gw *Gateway) NotifyCurrentServerStatus()
+func (gw *Gateway) ObfuscateKey(keyName string) string
+
func (gw *Gateway) ProcessOauthClientsOps(clients map[string]string)
ProcessOauthClientsOps performs the appropriate action for the received
clients it can be any of the Create,Update and Delete operations
func (gw *Gateway) ProcessSingleOauthClientEvent(apiId, oauthClientId, orgID, event string)
+func (gw *Gateway) ReplaceTykVariables(r *http.Request, in string, escape bool) string
+
func (gw *Gateway) RevokeAllTokensHandler(w http.ResponseWriter, r *http.Request)
func (gw *Gateway) RevokeTokenHandler(w http.ResponseWriter, r *http.Request)
@@ -9348,6 +9249,8 @@
Statuses of the request, all are false-y except StatusOk and
StatusOkAndIgnore
+func CtxGetRequestStatus(r *http.Request) (stat RequestStatus)
+
type ResponseCacheMiddleware struct {
BaseTykResponseHandler
// Has unexported fields.
@@ -9854,6 +9757,11 @@
Name() string
}
+type TykMiddlewareAsPlugin interface {
+ TykMiddleware
+ SetBase(*BaseMiddleware)
+}
+
type TykOsinServer struct {
osin.Server
Config *osin.ServerConfig
@@ -10123,9 +10031,8 @@
WH_DELETE WebHookRequestMethod = "DELETE"
WH_PATCH WebHookRequestMethod = "PATCH"
- // EH_WebHook is an alias maintained for backwards compatibility.
- // it is the handler to register a webhook event.
- EH_WebHook = event.WebHookHandler
+ // Define the Event Handler name so we can register it
+ EH_WebHook apidef.TykEventHandlerName = "eh_web_hook_handler"
)
type XPathExtractor struct {
BaseExtractor
@@ -10252,6 +10159,45 @@
func (t *TranslationFormatter) Format(entry *logrus.Entry) ([]byte, error)
+# Package: ./plugins/authKey
+
+
+
+CONSTANTS
+
+const (
+ ErrAuthAuthorizationFieldMissing = "auth.auth_field_missing"
+ ErrAuthKeyNotFound = "auth.key_not_found"
+ ErrAuthCertNotFound = "auth.cert_not_found"
+ ErrAuthCertExpired = "auth.cert_expired"
+ ErrAuthKeyIsInvalid = "auth.key_is_invalid"
+
+ MsgNonExistentKey = "Attempted access with non-existent key."
+ MsgNonExistentCert = "Attempted access with non-existent cert."
+ MsgInvalidKey = "Attempted access with invalid key."
+)
+
+FUNCTIONS
+
+func AuthFailed(m gateway.TykMiddleware, r *http.Request, token string)
+ TODO: move this method to base middleware?
+
+func GetPlugin(base *gateway.BaseMiddleware) gateway.TykMiddleware
+
+TYPES
+
+type AuthKey struct {
+ *gateway.BaseMiddleware
+}
+ KeyExists will check if the key being used to access the API is in the
+ request data, and then if the key is in the storage engine
+
+func (k *AuthKey) Name() string
+
+func (k *AuthKey) ProcessRequest(_ http.ResponseWriter, r *http.Request, _ interface{}) (error, int)
+
+func (k *AuthKey) SetBase(*gateway.BaseMiddleware)
+
# Package: ./regexp
package regexp // import "github.com/TykTechnologies/tyk/regexp" |
|
PR Description updated to latest commit (740aa9d) |
PR Review 🔍
Code feedback:
|
PR Code Suggestions ✨
|
💥 CI tests failed 🙈git-stateall okPlease look at the run or in the Checks tab. |
1 similar comment
💥 CI tests failed 🙈git-stateall okPlease look at the run or in the Checks tab. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
User description
Description
Modified the
mw_auth_key.gomiddleware to be compile-able as a Go Plugin, and have it loaded dynamically instead of as part of the monolith.Motivation and Context
This is a PoC to see if it would be possible to re-factor our middleware functions as dynamic plugins. If we were able to ship plugins as
.sofiles, it would be possible to ship fixes for specific middleware faster as we could simply provide patched.sofiles instead of full binaries.Secondly, having middleware as plugins would make it easier to ship enterprise plugins against the gateway, so we could differentiate paid / non-paid functionality.
How This Has Been Tested
Badly.
This is a PoC, it compiles, and it successfully loads the plugin in a demo API Definition.
Types of changes
PR Type
enhancement
Description
AuthKeymiddleware, to allow for more flexible deployments and updates.AuthKeymiddleware as a Go plugin, encapsulating its functionality and making it dynamically loadable.Changes walkthrough 📝
10 files
api.go
Export Internal Functions for External Usegateway/api.go
session, cache options, and more.
api_healthcheck.go
Export Health Check Reporting Functiongateway/api_healthcheck.go
reportHealthValuefunction to allow external usage.api_loader.go
Integrate Dynamic Plugin Loading for Middlewaregateway/api_loader.go
auth_manager.go
Export Token Generation Functiongateway/auth_manager.go
generateTokenfunction for external use.handler_error.go
Export Error Handling Functionsgateway/handler_error.go
errorAndStatusCodefunction.log_helpers.go
Export Key Obfuscation Functiongateway/log_helpers.go
obfuscateKeyfunction for external use.middleware.go
Export Auth Token Retrieval Functiongateway/middleware.go
getAuthTokenfunction for external use.mw_auth_key.go
Refactor Auth Key Middleware to Load as Plugingateway/mw_auth_key.go
mw_url_rewrite.go
Export Variable Replacement Functiongateway/mw_url_rewrite.go
replaceTykVariablesfunction for external use.mw_auth_key.go
Implement AuthKey Middleware as Go Pluginplugins/authKey/mw_auth_key.go
AuthKeymiddleware as a Go plugin.